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Abstract 


The goal of this project is to highlight the control methods of AI, based on the assumption that the Al 
will have consciousness and might constitute an existential risk. The report contains a segment of the 


ethical and moral issues that occur as a result of working within this field. 


To do so, Kant's definition of consciousness and variations of utilitarianism is used to establish a base 
perspective. It is through this perspective that the ethical issues that revolves around constraining a 


conscious Al are discussed. 


It was deemed fitting to look at the fictional portrayal of AI in visual arts, them being in the form of 
movies. This resulted in the choice of Ex machina, 2001: A Space Odyssey and Blade Runner. This 
analysis focuses on the ethical issues which arises from the implementation of the various control 


methods. 


We find that, from an utilitarian point of view which takes into consideration not only suffering and 
pleasure but also the quality of the pleasure, most of the control methods portrayed are seemingly 


either ineffective or unethical in their implementation. 
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1.0 Introduction 
and problem area 


* - 


The purpose of this chapter is to introduce the subject and premise of the report. First 
of all the introduction of the main themes, hereunder Artificial intelligence, singularity 
and the related control problem. After that, there will be an elaboration of the problem 
area, followed by the report's problem formulation. Secondly, the chapter will unfold 
the motivation for investigating this particular topic, which explains our own 


accademic and personal interest in the subject of artificial intelligence. 


n e 


1.1 Introduction & premise 


From an external point of view, choosing to create a project revolving around the philosophical 
issue of Artificial Intelligence may come off as unorthodox, if not unexpected. However, this 
may become less of a questionable choice, when observing the technological progress made by 


humanity throughout history. 


The question of Artificial Intelligence, may give rise to questions such as whether mankind puts 
itself above Nature or God. Earliest signs of this can be witnessed in the Roman Empire, as the 
city of Rome became one of the first metropolises and made progress in other technological 
areas, particularly in architecture. Their expansionist policy led the Romans to deforest a lot of 


areas, and in that sense, humanity already put itself above nature back then. 


Moreover, it was not until the Age of Enlightenment that humanity began to make substantial 


strides in its technological progress. 


At last, it was in the middle of the 20th century that Alan Turing, became the first person to 
make notable work on the field of AI, and gave name to the computer intelligence test known 


as the Turing Test. (Encyclopaedia Britannica Online, n.d.) 


Since then, humanity has continued the work done by Turing. At present, no AI exists that can 
match human intellect in a variety of areas. However, recent development have shown great 
progress, as current AI has not only surpassed humans in chess, but are also used to run the 
Google search engine. This year a new milestone was reached when the AI AlphaGo beat the 
Korean champion in Go, a game orders of magnitude more complicated than chess. (The 
Atlantic, n.d.) Looking at the progress made by humanity since Turing, it stands to reason that 
AI will eventually be improved upon and match, if not surpass human intellect in more areas 
than just board games. Based on simple observation on these past decades, if not centuries, one 


can tell that the technological level has continued to grow exponentially. 


Therefore, it seems prudent to write a project within this subject field as it does pose questions 
that humanity will need to address at some point. Considering the way that AI has been 
portrayed in fictional works, it also forces us to question the consequences of creating AI on par 
or above human intellect. Looking back at human history, numerous new technologies have 


been invented, seemingly without regard for the potential fallout. By creating AI, humanity will 


in a sense, give birth to a potentially superior race. When one considers the amount of warfare 
that has been waged throughout history, it is possible that the “child of humanity” will follow 


the footsteps of its parents. 


However, due to the current lack of ‘true’ AI, it is an uncharted field for humanity that would 
benefit from arduous study, particularly due to the fact that it will be the creation of a new 


intelligent entity, an attribute generally been seen as divine. 


Expectedly, this project will attempt to address those concerns to the best of the ability 


performed by the members of the group. 


To make certain that there is no discrepancy regarding the understanding of philosophy; we 
have chosen to utilize the definition given to us by Oxford Dictionaries, which defines it as “the 
study of the fundamental nature of knowledge, reality, and existence, especially when 


considered as an academic discipline.” (Oxford Dictionaries, n.d.) 


The pursuit of knowledge by humanity has led to research in AI, as previously mentioned. 
Creating AI will alter the reality and existence of the world, due to the emergence of a new 
species as a result of intentional works. Moreover, that species will be on par, if not superior to 
what is currently the most dominant species on the planet. Therefore, severe changes are to be 


expected. 


1.2 Problem Area 


As mentioned, this project will be based on the assumption that in due time, true AI will become 
a reality, and therefore it is imperative that we at least confront and accept that there will be 


problems caused by such emergence of a new power in the world. 


The state of research on AI today is not advanced enough to produce true AI, such as it is 
portrayed in works of fiction. For this reason we will be using fiction, specifically movies, in 


order to work with relatable representations of AI. 


Since we question the nature of the control problem, which is only necessary to impose upon 
strong AI, we will not be focusing upon weak AI. Strong Al is capable of functioning and solving 


problems across a vast number of different fields. Weak AI however, is only capable to work 


within a very limited field, usually only a single field. The definition of AI will be further 


expanded upon in chapter 2. 


In addition, we assume that the strong AI, will either develop, or have, some sort of 


consciousness. 
On AI Consciousness 


Seeing as AI will be acknowledged as a separate and artificial species, it begs the question as to 
how an artificial consciousness will function and the extent of it. We will aim to shed light on 


the subject, based on literary works. 


However, it is to be noted that our capability to handle this subject will be limited. This is a 
result of the current state of AI research. At this time, there are other matters in need of being 
addressed within this field, prior to engaging in research of artificial consciousness. Delving 
further into this area while lacking scientific knowledge and possessing limited theory, simply 


becomes conjectural. 


We will base our work on the assumption that the AI will have consciousness comparable to 


that of humans, which is why ethical issues are relevant. 


1.2.1 Problem formulation and work questions 


What is the nature of the control problem of AI, how does it relate to the control problem portrayed 
in science fiction and what ethical issues might arise if the AI is assumed conscious? 


Work questions: 


1. What has prompted philosophers such as Nick Bostrom, among others, to address the 


subject of Artificial Intelligence as well as presenting said subject as a problem? 
2. What is the control problem of AI and how is it portrayed in science fiction? 


3. Howcan the control methods portrayed in our chosen science fiction be employed 


without violating utilitarian ethics? 


1.3 Motivation 


As can be expected, the members of this group had various motivations for coming onboard 
this project. Some had already been studying this subject as a hobby inspired by the work of 
Bostrom, while others may have been primarily interested in the subject of Artificial 
Intelligence due to its frequent portrayal in fiction. Despite the level of entertainment provided 
to us by the media, we found it stimulating to apply philosophical knowledge to a practical 
problem, such as humanity’s future. As stated earlier, the technology of humankind continues 
to evolve exponentially, and the likelihood of superintelligent AI becoming a reality seem ever 


more plausible, which we of course will elaborate on. 


2.0 Theory and 
methods 


This chapter will introduce the theory and methods, chosen to approach this subject. 
First of all, the chapter will explain the choice of sources used in this report. Hereunder 
an explanation as to why we have selected the three films; Ex Machina, 2001: Space 
Odyssey and Blade Runner, for analysis. The chapter will also introduce the reader to 
singularity, artificial intelligence and superintelligence, which are essential terms in this 
report. The reader will also be properly introduced to Kant’s definition of consciousness 
along with utilitarian ethics. The chapter will end with the epistemological 
considerations underlining the report structure and an elaboration of the analysis and 
discussion method used in chapters 5 and 6. The end of this second chapter will present 


a critical reflection on the process of working with this project. 


- 


2.1 Choice of Theory and Sources 


In every academic work ever made, it is of utter importance to ensure that the integrity of the 
sources are verifiable. After all, they do set the foundation for the work itself and essentially 
become the pillar supporting the findings of scholars and the like. Therefore, we have chosen 


to make use of criticism of sources as presented by Steven Bailey (2014: page): 


(a) What are the key ideas in this? 
(b) Does the argument of the writer develop logically, step by step? 
(c) Are the examples given helpful? Would other examples be better? 
(d) Does the author have any bias (leaning to one side or the other)? 
(e) Does the evidence presented seem reliable, in my experience and using common sense? 


(f) Do I agree with the writer’s views? 


Naturally, it would be a project in itself to apply these six points to all the content within the 
sources listed in the bibliography. It could be done to ensure, with absolute certainty, that the 
authors maintain the consistent quality throughout their works, so we will simply summarize 


our approach. 


First of all, our choice of sources were primarily due to the authors being notable in that field, 
that being AI or philosophy of mind, therefore they would have a level of legitimacy. Moreover, 
we looked into the works of several authors in order to compensate for any potential issue there 
might have been with using a single one, which could be done by finding a middle ground 
between the authors or analyze the quality of the work by the sources. While some of us had a 
background in natural sciences, others in the group came from the humanities. Regardless of 
the background, we are already acquainted with the process and ways of making fallible and 
infallible arguments. Again, having multiple sources within the same field allowed us to negate 


any possible argument that did not arise step by step. 


Whether one agrees with the writer's views may be dependent on the writer himself. In 
Bostrom's case, he argues from multiple perspectives in order to account for any potential 
criticism, or at least that is the obvious reason. He provides not only helpful examples, but 


presents them in a clear and concise way that is not convoluted, granted they easily could be. 


However, there is the matter of using a secondary source on Kant and his ideas on 
consciousness. Some of the difficulties that may have sprouted as a result of that choice, could 
be translation errors. Without having read the original untranslated works of Kant, it is hard to 
determine the accuracy of the translation. In addition, we were unable to discover any reviews 


of the book, except for an editorial review, which could be argued to be biased in nature. 


Regardless, the book does have over a dozen editors as well as three main editors, which should 
have decreased the margin for error, since their work is based on the quality of the various 
translations. Seeing as we only made use of the section on Kant, there is not really anything to 


note other than it following the standard formula as listed above. 


Our definition of consciousness comes from the editorial review book's chapter on Kant. 


2.1.1 Movies used in this project 


The approach we are using when dealing with movies is based both on cinema semiotics and in 
aesthetic modernity. From semiotics we are going to take the relevance of the ‘meaning’ and 
from aesthetic modernity the importance of the images. However, we do not completely follow 
the previously mentioned methods, as we are going to let the scenes from the movies ‘talk’ as a 
basis for the developing of our critical discussion. We will start out this part by presenting a 
general overview about them, and then analyzing the problems we want to reflect through the 


images. 


When dealing with AI we found that, as it is something that has not arrived yet, it was difficult 
to present scenarios and situations in a comprehensive way. For this reason, we thought that 
using science fiction as a basis to discuss upon was a good idea, particularly because it 
presented the material that we wanted to analyze and discuss. It is clear that science fiction is 
mostly speculative, but when dealing with issues that may come in the future there is no other 
choice but to take into account this kind of data, that may give us clues to reflect on what is yet 


to come. 


However, science fiction is presented in different media, and during the initial phase of the 
project we thought about using both films and literature. In the end, we decided that the 


scenarios we wanted to portray were better represented through visual meanings. Instead of 
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completely erasing science fiction literature off the list, we chose to use it as an inspiration for 


our project and focused our work on the selected movies. 


When choosing the movies, we wanted to have a wide spectrum of different Als portrayed and 
a reasonable quantity of material to work with at the same time. We also wanted works 
produced in different contexts and times. Our final decision was to analyze 2001: A Space 
Odyssey (1968), Blade Runner (1982) and Ex Machina (2015). The choices were not random. 
The three films present wide differences in the kind of AI, the perspective on it as well as the 
control methods used. The movies were also produced in different times, covering almost fifty 
years of science fiction production which guide us through the different ways AI was perceived: 
from the optimism regarding the possibility of creating an AI of the sixties until the recent 
realization of the difficulties that this presents. Leaving those reasons apart, all of them are 
works with an indisputable cinematographic value and, specially 2001: A Space Odyssey and 


Blade Runner, have been vastly discussed in academic grounds before. 


It should be noted that none ofthe films we have chosen portrays a version of AI with a level of 
superintelligence that would constitute a decisive strategic advantage. As such the Als we 
analyze have not become existential risks yet. The reason why we did not choose science fiction 
where the AI has a decisive strategic advantage, such as the Terminator series or the Matrix, is 
thatthose movies do not concern themselves with the AI as a conscious being with moral status. 
On the other hand, the movies we have chosen emphasize the human-like nature of the Als 


while suggesting that the AI needs to be controlled in some way or another. 
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2.1.2 Theory used in this project 


Approaching the ethical issue 


“Schools of ethics in Western philosophy can be divided, very roughly, into three sorts. The first, 
drawing on the work of Aristotle, holds that the virtues (such as justice, charity, and generosity) 
are dispositions to act in ways that benefit both the person possessing them and that person’s 
society. The second, defended particularly by Kant, makes the concept of duty central to morality: 
humans are bound, from a knowledge of their duty as rational beings, to obey the categorical 
imperative to respect other rational beings. Thirdly, utilitarianism asserts that the guiding 
principle of conduct should be the greatest happiness or benefit of the greatest number” (Oxford 


Dictionaries, n.d.) 


That is how ethics is defined according to Oxford Dictionaries. In respect to our ethical 


perspective we decided to choose utilitarianism for mainly three reasons. 


The first is the above-listed definition, which should allow no room for misunderstanding on 
the word ethics, not to mention that it is an unbiased and external source. Second of all, 
utilitarianism can be seen, altogether with deontological theories based on Kant’s work, as one 
of the most used approaches in contemporary ethics. Therefore, utilitarianism has a solid 


background that makes it appropriate to work within any academic research. 


The third reason is that we wanted to use a theory that could be applicable to every part of the 
project, not only to humankind's ethical views about the AI and its control methods, but also as 
a perspective that an AI could easily have. An approach in ethics such as the Kantian one, 
previously mentioned, would present much more problems: the principle of universalizability, 


for example, could differ greatly when applied by Als, as their values may be radically different. 


The balance between pleasure and suffering, altogether with the well-being of sentient beings, 
appeared as the easiestone to translate to computing language, given its quantifiable character. 
That, however, does not mean that it is a simple task and, as we will argue further, translating 
terms such as ‘pleasure’, ‘suffering’ or ‘well-being’ can be complicated. It gets even more 
difficult with Mill's qualitative variant of Bentham's classic utilitarianism. Even so, 


utilitarianism, dealing with the previously mentioned variables, make us think that its use in AI 
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research may be more applicable, specially when the free will of these subjects is not easily 


probed. 


Perspective on consciousness 


We chose to use Kant's perspective on consciousness, as it is based on the premise that one 
would fulfill certain criteria set forth by Kant. These criteria range from being able to reflect 
upon one's own abilities, to how one can form new truths from old and new information. By 
choosing to use Kant’s perspective, we evade the long discussion in a muddled theoretical field 


of what a consciousness is. 


Understanding AI, superintelligence and the control problem 


In the initial phase while working on this topic, we wanted to completely define and understand 
the underlying themes of the subject. We found that a mutual understanding would contribute 
to a coherent report. To help us with that, we chose two contemporary philosophers; Nick 


Bostrom and David Chalmers, as our primary source for understanding these themes. 


Nick Bostrom is Professor of philosophy at Oxford University and founder of The Future of 
Humanity Institute, as well as The Programme on the Impacts of Future Technology within the 
Oxford Martin School. He is, in a sense, the leading authority on the topic of control problem in 
AI. In 2014 he published the book "Superintelligence: Paths, Dangers, Strategies”. The book, is 
the first of its kind, and explores the dangers of superintelligence, from a philosophical point of 
view. We will use this book, in understanding and defining superintelligence, which will be 
applied to the chosen movies, answering in what way the Als in the movies are to be considered 
superintelligent. Later on (in chapter 4) we will elaborate on the control problem and Bostrom’s 


suggested solutions for that matter will be applied to the analysis of the chosen movies. 


The philosopher David Chalmers specializes in topics regarding the mind and consciousness. 
We made the choice of Chalmers as a primary source, as he is broadly referred to regarding the 
subject of AI. We mainly used his 2010 article “The Singularity: A Philosophical Analysis". Like 


Bostrom, the text provided us with a philosophical approach to the subject, thereby the choice 
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was straightforward. Chalmers text lead us to the understanding of singularity, and in doing so, 


he also helped us in our definition of Artificial Intelligence. 


As we dug into these themes, we found a lack of clear definitions of the used terms. The 
definitions were either implied, very broad, or just vague, and therefore we found no immediate 
use for them. As a result, we chose to create a working definition of AI and superintelligence. 
These definitions will be found accordingly after the introduction of Bostrom and Chalmers’ 


understanding of the themes, and will be our contribution to the used theory in this report. 


2.2 Utilitarianism & Kant 


2.2.1 Utilitarianism 


This part will deal with the ethical perspective used in this project. In the following chapters we 
will elaborate further on the problems surrounding control methods and consciousness, but 
the question that we want answer in this report is: ‘Is it ethical to attempt to control or 
manipulate a being that can have consciousness and intelligence in the same or even in a 


superior way to the one that humans have?’ 


To answer this question, we are going to apply utilitarianism to the subject. In order to do that, 
we will proceed to give an explanation of the theory and of the concepts and principles that we 
are going to use. We will start with Bentham, the one who first formulated utilitarian theory. 
After that, we will present the development that Mill’s work involves, as well as adding some 


concepts from Singer. 


Jeremy Bentham's Theory of Utilitarianism 


The first thing to understand before venturing into utilitarianism is that it is a theory that bases 
its ethics in the consequences of an action, not in the intention nor the act itself. It is therefore 
a consequentialist theory. As Bentham puts it in his major book An Introduction to the 
Principles of Morals and Legislation, first printed in 1780, “the general tendency of an act is 
more or less pernicious, according to the sum total of its consequences: that is, according to the 


difference between the sum of such as are good, and the sum of such as are evil” (1780: 61). 
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The following question that appears is ‘what makes the consequence of an action good or bad?’ 
For Bentham that is what he calls the principle of utility. For ‘utility’ Bentham means everything 
that “tends to produce benefit, advantage, pleasure, good or happiness” (Ibid: 14). Then, those 
acts whose consequences follow this principle would be desirable, and those that generate the 
contrary pernicious. Those acts that reduce the amount of undesirable consequences, such as 
pain, will also be considered good. The balance of the consequences of an action would be 
measured in a series of circumstances or properties of pleasure and suffering: intensity, 
duration, certainty, propinquity, fecundity, purity and extent. However, we will not dwell any 


further on this complex calculation. 


This is applied to the ‘interest of the community’. Taken the above into play, that means that 
the goodness or badness of an action is measured by the “the sum of the interests of the several 
members who compose it [the community]” (Ibid: 15), meaning with ‘interest’ the avoidance of 


suffering and the obtaining of pleasure. 


It is to be noted, that the concept we are always using is ‘action’. Bentham talks about rules, 
although those are not moral rules but ‘measures of government’, that is a particular type of 
action: laws created by the governors. For Bentham, then, the principle of utility is the maxim 
for every single action and all the acts that contribute positively to the pleasure/suffering 
balance should be performed. This, as we are now going to explain, differs from the conception 


by Mill. 


John Stuart Mill’s develop of Bentham’s utilitarianism and Nozick’s criticism of the 


theory 


Mill, though following Bentham’s work, introduces some important variations in the theory. He 
takes another direction and his utilitarianism is more qualitative than quantitative. Pleasure 
and suffering are not measured using Bentham’s properties, but in terms of ‘quality’. This 
‘quality’ is what makes an individual to place one pleasure above other ones “irrespective of 
any feeling of moral obligation to prefer it” (Mill, 1863: 16). Even when Mill’s theory is still 


aggregative it gains richness compared to Benthams', making values come into scene. As he puts 
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it: “It is better to be a human being dissatisfied than a pig satisfied; better to be Socrates 


dissatisfied than a fool satisfied” (Ibid: 19). 


This notion of pleasure and suffering refutes Nozick’s posterior critic to utilitarianism theory. 
In Anarchy, State, and Utopia Nozick proposes the thought-experiment known as The 
Experience Machine. The exercise is quite simple: imagine a machine that would provide all the 
pleasant experiences that one could desire, making feel to the connected ones that what they 
live is the reality, would the people let themselves get plugged in? Nozick argues how most of 
the individuals would decide not to plug in, because living the reality is more important to them 
that the pleasures that the machine can provide. Pleasure and suffering, then, should not be the 


terms in which the moral is measured. 


As we see it, Nozick here uses Bentham’s utilitarianism to criticize all the utilitarian theory. 
Taking the qualitative notion about pleasure described by Mill, we can discuss that the reason 
why most of us would not plug in the machine is because living the reality is a pleasure of higher 
quality than the experiences that the machine can offer. Choosing to live outside the Matrix is 
not a choice which rejects pleasure, but it is a choice which chooses the pleasure of higher 
quality. As the decision is taken in the reality, even when you know the experiences will look as 
reality since connected, the decision of plug in is not usually taken. However, we can also argue 
that some would connect themselves to the machine: “Men often, from infirmity of character, 


make their election for the nearer good, though they know it to be less valuable” (Ibid: 19). 


Mill’s utilitarianism, in opposition to the one defended by Bentham, can be considered as rule 
utilitarianism, that is, the morality should not reside in the concrete action, but in a moral rule 
that, when followed, provide happiness to the majority. In that way, extreme acts, such as killing 
an innocent for providing pleasure to a large number of persons, are avoided. However, even 
when morality should be composed by these rules, some of them have more priority than 
others: “Thus, to save a life, it may not only be allowable, but a duty, to steal, or take by force, 
the necessary food or medicine, or to kidnap, and compel to officiate the only qualified medical 
practitioner” (Mill, 1863: 113). That is because the moral rule of saving a life is more important 


than the one that tells you not to, for example, steal. 
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Equal Consideration of Interests and Speciesism 


The ‘principle of equal consideration of interests’ is quite relevant in utilitarianism. Based on 
what Mill calls ‘Bentham’s dictum’ (“everybody to count for one, nobody for more than one” 
(Mill, 1863: 112)), the principle was further developed by Richard Ryder and later popularized 
by Peter Singer in what they call ‘speciesism’. The idea of speciesism is, basically, to “give 
greater weight to the interests of members of their own species when there is a clash between 
their interests and the interests of those of other species” (Singer, 1993: 58). Critique of 
speciesism has mainly been used to defend the interests of animals, arguing that animal 
suffering is also to be taken in consideration. That is not to say that human and animal suffering 
are the same, because human and animal mental capacities differ in terms of anticipation, 


memory, knowledge or future planning, to name some examples. 


Speciesism, however, can also be brought into play when dealing with non-organic species, as 
long as their capacities allow them to experience pleasure or suffering, among other things. This 


can get even more complicated when dealing with consciousness in AI. 


2.2.2 Kant and consciousness 


The idea of an artificial intelligence having consciousness is still quite abstract, but as our 
technology advances and, perhaps with the help of artificial neural networks [see 2.3.4], the 
possibility of the construction of an artificial consciousness seem to become more likely. In this 
report we will be discussing theoretical ways proposed to impose a certain amount of control 
upon an AI. As such, if the AI in question has a consciousness on par, or at least remotely 
relatable to that of a human being, on which moral ground can we in good faith impose 


sanctions upon another being with consciousness? 


However, before we can begin this discussion, we must first define what we mean when we talk 


of consciousness. 


The theories and ideas of what consciousness is, has been a widely debated topic for hundreds 
of years. In spite of this, humanity have yet to come to a clear definition that can be universally 


agreed upon. Regardless, throughout the long debate several interesting and intriguing 
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positions have been made. As the title of this chapter suggests, our choice for looking and 


defining consciousness comes from Immanuel Kant (1724-1804). 


Kant sets up parameters which must be fulfilled prior to making any claims of an existing 
consciousness, which we will elaborate on in the following section. The advantage of using such 
a specific definition, which requires that certain criteria be met, is that it is a pre-set set of tools, 
which is easy to utilize in an analysis and use these to describe certain characteristics the AI in 


question should have. 
Immanuel Kant’s definition of consciousness. 


Kant argues, that in order to claim that a subject has consciousness it must be able to fulfill 
certain criteria, as mentioned in the previous section. These criteria can be narrowed down to 
how a subject perceives the world around it, or the subject choosing to reflect on what it has 


perceived. (Heinämaa, 2007) 


The terms ‘a priori’ and ‘a posteriori’ serve as an important structural part of Kant's theory. The 
‘a priori’ (Ibid.), that which is required in order to make empirical recognition possible, and the 
‘a posteriori (Ibid.) , that which is based on the recognition. Using these terms creates an 
important distinction between ‘That which is required before an observation can be made’ (a 


priori) and ‘That which is a conclusion based upon said observation’ (a posteriori). 


The a priori offers certain parameters and ways of looking and defining consciousness as, an 
intelligence which can and will interest the being itself (at least to some degree) in; knowing 
and reflecting on not only what it perceives through it senses, but also how its senses function 
and how it could know anything about its own abilities to make observations. Therefore, when 
using Kant’s a priori, a conscious artificial intelligence will have the ability to question both how 
it is perceiving and how it is reflecting upon the object (the world) which surrounds it. This 
definition of an AI's consciousness concerns itself with knowledge and learning, how a machine 
is both capable of acquiring new knowledge on its own accord and, more importantly, the way 


it chooses to ask itself how it is able to gather this knowledge. 


Another important term to mention is Kant's transcendental account (Heinämaa, 2007). Kant 
speaks of transcendental knowledge, that which concerns itself with how one is able to 


experience something as an object. Not only that, it also deals with the perceiving of said things 
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and objects as not just physical entities, but as metaphysical entities. Therefore, Kant’s 
transcendental knowledge also recognizes objects as entities that supersede physical 
boundaries, having meaning beyond mere physical existence. However, the mind must be 
capable of creating a link between the physical appearance of the object, and the idea of the 


object. 


Yet another important term Kant introduces is the transcendental reflection (Ibid: 236), which 
is how to reflect on cognitive reflection of the representations. How to use old and new 
knowledge and how, through reflection, being able to determine under which circumstances 
the cognitive judgement would be correct. It is an object-based type of judgement, concerning 
itself with how we, via knowledge ofa (or more) object(s), can make predictions and reflections, 


and set up certain parameters which must be met before these predictions can be true. 


In the context of the report, this transcendental reflection can help us determine the extent to 
which the consciousness of an artificial intelligence and that of a human mirror each other, 
seeing as it provides a parameter for measurement, the ability to reflect i.e. how does one make 
sense of and use old 'representations' and 'new representations' to measure if, and under which 


circumstances a given concept of intuition can be given any validity at all? 


Through the use of this concept, we are given a tool to determine how an AI would use its older 
data, along with new data, to formulate hypothetical scenarios. Moreover, it would also be able 
to utilize the data to predict the likelihood of said scenarios, as well as the parameters required 


to enable the possibility of those. 


One last concept Kant introduces is the anti-skeptical premise (Heinàmaa, 2007), which states: 
“Tam conscious of my own existence as determined in time" (Ibid: 236). The use of this concept 
presupposes that the subject should be capable of perceiving and reflecting upon its own 
existence, in order to be considered conscious. What this concept adds is that the subject should 
consider its existence as finite in a timely sense. The subject's existence is not infinite, it had a 
starting point, it exists in the present, and it will have an end. This concept concerns itself with 


how a subject perceives its own existence in the world, measured in time. 


One important question which could be raised when using this concept, could be how an AI 


would be able to perceive its own evolution through time, i.e. its accumulation of knowledge. 
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Would it be able to ‘remember’ its existence prior to achieving consciousness? Will it be aware 
of the fact, that it might not be immortal, and how will it react to the revelation, that it is going 


to someday cease to exist as a thinking operating subject? 


Kant's four concepts will be the parameters for how we define and work with the idea of an AI 


consciousness. The four concepts are as follows: 


1. ‘A priori’ and 'a posteriori’ - It must be able to have the conditions needed to make 
perceptions; the ' a priori' constitutes the condition needed by the ' a posteriori' in 


order to perceive. 


2. Transcendental account - It must be able to perceive objects as more than mere 


physical existence, thereby having the capability to reflect abstractly on the objects. 


3. Transcendental reflection - It must be able to utilize both abstract reflections and 
perceptions in conjunction, in order to create new independent reflections and 


perceptions placed in time and space. 


4. Anti-skeptical premise - It must be able to perceive itself as a thinking existence which 
has existed before the present, which exists in the present, and will continue to exist 
past the present. The AI must be able to understand itself as a finite being, that have a 


start and will eventually have an end. 


2.3 The singularity, AI, and superintelligence 


This project revolves around the topic of Artificial Intelligence (AI), the singularity and the 
possible control problem resulting therefrom. With that in mind, the following section will 
properly introduce the given terms, and thereby support the reader with a general 
understanding for reading this report. More specifically, the chapter will introduce and 
elaborate the terms AI, superintelligence and the singularity. We will follow up on them with 
our definition of the terms. In conjunction with introducing the terms, this chapter will also 
elaborate on the premise of the project. As mentioned in the introduction and problem area, 
the relevance of this project is dependent on the premise that, eventually, there will be Artificial 


Intelligence (AI). As writers ofthis project, we are therefore in need of underlining the accuracy 
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of this premise and thereby emphasize the significance of the problem presented by this 
project, in terms of intellectual expansion, interest by philosophers of mind and the existential 


threats and benefits associated with the issue. 


2.3.1 Defining artificial intelligence 


The definition of AI may seem simple, as it lies in the meaning of the word that Artificial 


Intelligence is intelligence that is - artificial. It is generally referred to through its acronym AI. 


Since humans set the bar for intelligence and Als will in fact be compared to human intelligence, 
we first considered using Encyclopaedia Britannica’s definition which is: “Mental quality that 
consists of the abilities to learn from experience, adapt to new situations, understand and handle 
abstract concepts, and use knowledge to manipulate one’s environment.” (Encyclopaedia 


Britannica Online, n.d.) 


However, in the process of writing this project, we discussed AI and found that this definition 
was not that simple or operational. To understand and support the premise this project is 
founded upon, we are in need of a workable definition of the term AI. This clear definition will 
come in handy for the reader, as well as for us (the writers of the project). In that context, the 


upcoming section will explain the definition of AI used in this project. 


In this project, our primary literature about AI is by David J. Chalmers and Nick Bostrom. In 
some way, both of these writers refrain from defining the term itself. Therefore we chose some 
secondary literature by Russell & Norvig (1995) that can help us with an applicable definition 


of the term. 
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The common perception of AI 


A common perception of how to detect AI is by the use of the Turing test. If a machine passes 


the test, the machine is AI. 


The Turing test was designed to contribute to a "satisfactory operational definition of 
intelligence" (Russell & Norvig 1995: 2). The test was a part of Turing's paper Computing 
machinery and intelligence, published in the journal of philosophy Mind in 1950 (Turing, 1950). 
The main goal was to propose a test for detecting AI. Instead of making a computer generated 


test, he proposed a test with a human interrogator. The test itself goes as follow: 


the interrogator asks the computer and a human control some written questions, and gets a 
written response. The test is passed if the interrogator is unable to tell if the responses come 


from a person or not. If the test is passed, the intelligence is AI. 


The test is understood as a common perception, as the test is broadly referred to in both fiction 


and academic literature. 


As Russel & Norvig explains, one can discuss if a computer is "really intelligent", just by passing 
the Turing test (Russell & Norvig 1995). In order to understand the foundation of this 
discussion, you can say that the Turing test only investigates whether or not the system acts 
like humans, and thereby it defines AI as a machine that acts human. In this context we will 
refrain from delving further into this discussion, but it is worth mentioning that this 
understanding (that a machine is AI, if it acts human) is not shared by all stakeholders in this 


topic, and that there are several perceptions of the term. 


Russell & Norvig found that the way AI is defined differs in two main dimensions, the ones 
concerning thought processes and reasoning, and the ones concerning behavior (Russell & 
Norvig 1995). With these two dimensions in mind, Russell and Norvig explains that the 
definition of AI can be divided into four categories, which can be illustrated by the table on the 


next page. 
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A philosophical approach to the control problem of artificial intelligence 


Systems that think rationally 


“The study of mental faculties through te use of 
computational models.” (Charniak and McDermott, 
1985) 


“The study of the computations that make it possible to 


perceive, reason, and act.” (Winston, 1992) 


Systems that act like humans 


“The art of creating machines that perform functions 
that require intelligence when performed by people.” 


(Kurzweil, 1990) 


“The study of how to make computers do things at 
which, at the moment, people are better.” (Rich and 
Knight, 1991) 


FIGURE 1: OWN REPRESENTATION, OF “THE FOUR DIFFERENT AI APPROACHES” (RUSSEL & NORVIG 1995: 4-8) 


These four different approaches are all relevant in the understanding of what Al is, as they show 
that the definition of the term varies. It can be argued that a single definition of AI is non- 


existing at this point. 


Our definition of AI 


The definition of the term should be understood in the context of this report only, and we do 
not suggest to use this definition as a general understanding of AI in all other contexts. We 


suggest a definition as follows: 


Artificial Intelligence is a human-created system, whose behavior and thinking patterns is 


comparable or better in capability with that of the average human being. 
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This definition points out that AI should be human-created. By that, we mean the opposite of 
something that has evolved without human interference. In theory, the AI could also be 


machine-created (second order human creation), which still refers to not being evolved. 


AI should be human-like in both thinking and behavior. We claim this based on some of the 
perceptions mentioned in the previous table [figure 1]. Our definition differs, as it suggests that 
AI should be comparable with human both in thinking and behavioral patterns. This implies 
that the Als discussed in this project are ‘true’ AI. That is, AI possessing some level of 


consciousness. 


As the definition also claims, AI should be at least comparable with the average human being. 
This statement will perhaps make more sense in the next section where we elaborate on 
superintelligence. But to explain this, one could say that AI is somewhat equal to human 


intelligence (opposed to superintelligence, which is more intelligent than human). 


2.3.2 Superintelligence (AI++) 


This project revolves around the control problem of AI. We previously defined AI in the 
preceding section and noted that it is comparable with the average human being. Therefore it 


is not here the control problem lies. 


As Nick Bostrom's book (and the title of the same) suggest, the control problem lies in 
superintelligent AI - or as Chalmers defines it - Al++. Al++ is defined by Chalmers as the future 


result of the singularity (term further explained later in this chapter). 


What do we mean by superintelligence? The following definition is inspired by the one Nick 
Bostrom present in Superintelligence: Paths, Dangers, Strategies. He divides superintelligence 
into three categories; speed superintelligence, collective superintelligence and quality 
superintelligence. The segmentation is relevant, but as Bostrom himself explains, they are (in 
some way) equivalent (Bostrom, 2014). The need for a definition in this report is, linked to the 
need for mutual understanding, therefore we find no need to use the separation in this context. 
Instead, we will combine the three types, thereby creating a general definition of 


superintelligence. 
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The definition of superintelligence 


The general definition of superintelligence is an entity more intelligent than human, meaning 


that the superintelligence should outperform humans. 


As Bostrom states, we do already have examples of superhuman powers in the present world. 
This can be observed in animals that outperform humans physically, e.g. bats, which navigates 
using sonar signals, or machines that outperform humans in logical reasoning, e.g. calculators 
that can solve arithmetic (Bostrom, 2014). Although humans are outperformed in some 
domains, we have none (machines or animals) superior to human in every aspect. On that note, 
the definition of superintelligent beings could more accurately be referred to as; “(...) intellects 
that greatly outperform the best current human mind across many very general cognitive 
domains.” (ibid. 52). Bostrom’s definition of superintelligence claims many different statements 
explaining how an entity can be considered superintelligent. With this information in hand, the 


next section will elaborate on our definition of superintelligence. 


Our definition of superintelligence 


In the formulation of a common understanding of superintelligence, the definition becomes 
quite vague. In order to give an (as exact as possible) understanding of the term, the following 


definition will therefore consist of four statements for superintelligence. 


Four statements for superintelligence: 


§ 1: Superintelligence should outperform the best current human mind in many areas 
§ 2: Superintelligence’s overall performance should be superior to that of humans 
§ 3: Superintelligence should be at least as fast (or faster) than human intelligence 


§ 4: Superintelligence should be as superior to human intelligence as human intelligence 


are superior to the intelligence of a chimpanzee 
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Elaboration of the four statements: 


$ 1: Superintelligence should outperform the best current human mind in many areas 
This statement links to the previous example with the calculator. Even though the 
calculator outperforms us in solving arithmetic, we easily outperform the 


calculator in e.g. social intelligence 


§ 2: Superintelligence’s overall performance should be superior to that of humans 
This statement refers to the Nick Bostrom’s collective superintelligence. The 
choice of word ‘overall’, ties that a superintelligence could consist of many 


intelligences, which (only) combined would outperform human. 


§ 3: Superintelligence should be at least as fast (or faster) than human intelligence 
The third statement relates to Bostrom’s speed superintelligence, which he 
formulate as “A system that can do all that a human intellect can do, but much 
faster" (Ibid. 52-53). The formulation “much faster”, does not preset the magnitude 
of the speed, which is why the formulation is not used in this statement. This 
statement also makes room for the superintelligence to not (necessarily) be faster 
than human, but just as fast, this refers to Bostrom’s definition of quality 
superintelligence, where he states that this kind of intelligence is at least as fast as 


human mind (Ibid.). 


$ 4: Superintelligence should be as superior to human intelligence as human intelligence is 
superior to the intelligence of a chimpanzee 
This statement is the only statement that tells us about the magnitude of the 
intelligence in superintelligence. The statement could also go; "superintelligence 
should be x times as intelligent as human intelligence", but as we have no 
knowledge of how much the intelligence should exceed human intelligence, that 


definition would be inaccurate. 


The example with the chimpanzee comes from Nick Bostrom's definition of quality 
superintelligence, which give us an analogical understanding of the magnitude of 


intelligence in superintelligence. 
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As mentioned in our definition of AI, we assume that a true Al's thinking and behavioral 
patterns should be comparable to a human. Both Bostrom and Chalmers claim that it is not 
necessary for AI to have a consciousness, though from an ethical perspective, it is not only 
logical, but also necessary to assume that the AI and superintelligent AI we are discussing have 


some sort of consciousness. 


Since we cannot know if a future AI will have consciousness, it seems prudent to at least include 


that possibility of this in the discussion of the control problem. 


2.3.3 Singularity - why there will be superintelligence 


LJ. Good explains that superintelligent machines will leave man behind and that "the first 


ultraintelligent machine is the last invention that man need ever make" (Good, 1965). 


I.J. Good, the author of the term we now know as ‘the intelligence explosion’, stated this back in 
1965 and it is still used as a way to easily explain the singularity (e.g. by Chalmers, 2010). To 
elaborate this statement, we will now explain the fundamentals of the hypothetical 
phenomenon technological singularity, which will be referred to as the singularity from now 
on. The term singularity, was introduced by Verner Vinge in a 1984 article, but was first 
widespread by his article "The Coming Technological Singularity" from 1993, and by futurist 
Ray Kurzweil's book "The Singularity is Near" from 2005 (Chalmers 2010). 


Singularity is the event where humankind invent a machine that is more intelligent than man 
(an AI+). This machine (AI+) will be able to create more intelligent machines than man (AI+). 
These intelligent machines will, in turn, create even more intelligent machines (AI++). It is also 


called the intelligence explosion as mentioned earlier. 


Theoretically this will continue until we get an infinitely intelligent machine, which is a difficult 
idea to grasp and may not even make sense. In reality it will most likely encounter a limit before 


reaching infinity, nevertheless it will expand and do so at an alarming rate. 


The terms AI, Al+ etc. will be used to differentiate the generations of AI, and thereby the degree of 
intelligence (AI being the first generation of intelligence, and Al* the next and Al** being 


superintelligence). 
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The intelligence explosion is often thought as going hand in hand with the speed explosion, 
which is deduced from the observation that the processing power doubles every two subjective 


work years, known as Moore’s Law. 


Suppose then it does so indefinitely, and that we have human-level artificial machines designing 
the processors. Using the processors will allow the designers to function faster, making the 
regular interval of two years irregular and a lot shorter, resulting in a very fast design cycle and 
shortly thereafter a limit. This limit could theoretically have any magnitude but it would be 
finite. Fortunately, we have some physical restraints that are a bit easier to understand than 


that for infinite intelligence. 


These two explosions are logically separated, but the speed explosion is very likely to follow 
the intelligence explosion, though it is not very likely that the intelligence explosion follows the 


speed explosion. 


It is more likely one way than the other, because a superintelligent machine might realise that 
it can improve drastically by sparking the inception of a speed explosion alongside the 


intelligence explosion. 


Many different sources stipulate that the AI will come sooner rather than later (within a few 
decades as opposed to hundreds of years). Chalmers argues that most of them state this with 
Moore’s Law in mind, which tells us something about the advancement of hardware, but the 


biggest bottleneck is software, not hardware (Chalmers, 2010: 6). 


2.3.4 The use of neural networks in AI 


As claimed in "Our definition of AI" [page 23], AI should be at least human-like in both thinking 
and behavior. Such properties are indeed challenging to implement in the AI. A promising 
approach for developing AI is by the use of artificial neural networks (ANN). This section will 


shortly introduce the concept of such artificial neural network. 
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A short introduction to Artificial neural networks (ANN) 


The evolution of artificial neural networks began in the 1940s as an attempt to understand the 
human brain, as well as imitate some of its strengths (Kattan et al. 2011). Since then humanity 
has gained more knowledge about the brain. The combination of that and the increasing 
computer power has fostered the development of ANN with more and more application- 


possibilities (Zhang, 2010). 


As mentioned, ANN is an attempt to imitate the human brain. The human brain is a (non- 
artificial) neural network , which consist of neurons and axons. The neurons are connected by 
the axons, and signals are sent from each neurons, thereby affecting the whole network. The 
neural network provide us with a unique learning capability, where our brain learns from the 
experience of every neuron. Understandably, this learning capability is very attractive for Al 
designers, which is why the implementation of artificial neural network in AI is a very popular 
approach (Luger, 2005). To this day artificial neural networks are (amongst other) used as tools 


for face-recognition and for controlling Google's self-driven car (Simonite, 2012). 


An example of ANN: 
Hidden 


Input | | 


FIGURE [2]: AN EXAMPLE OF AN ARTIFICIAL NEURAL NETWORK (ANN) 


To each node (depicted as a circle) there is a weight and a sum of arrows connected to it. The arrows signify the 
direction of the in- and outputs and the weight helps determining whether a node fires or not (goes 1 or 0). The 
firing ofa node is determined by both the weight and the algorithm used by the node, which combines the inputs 
from either the source or other nodes, where after it either fires or does not. The weights can be adjusted by using 
learning methods. 
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On the previous page, was a picture of a double layered feedforward network, which is some of 
the most simple examples of an ANN. The only important parts regarding ANN for this project 
is that it learns and does not have everything it knows programmed (as in predefined) and that 
at the current state of technology it seems probable that we end up with an AI based on this 


type of system. 


Why use neural networks in AI? 


As elaborated in our definition of AI [page 23], AI should have behavioral and thinking patterns 
comparable or superior to that of the average human being. ANN has shown itself to be very 
good at pattern-recognition, due to ANN being able to detect regularities in large amounts of 
data, which would be almost impossible using other calculation models. (Shiffman, 2012) 
Human behavioral and thinking patterns are very complex, however, ANN is probably the most 


effective solution for imitating these patterns in the creation of AI. 


2.3.5 Singularity as an existential risk 


We have learned that superintelligence (AI++) is a result of the singularity, and that it can bea 
reality through the use of neural networks. Nevertheless, what is the problem in that? Nick 


Bostrom presents the singularity as an existential risk, which we will elaborate on. 


Bostrom states the arguments for fearing an existential catastrophe as a default outcome ofthe 
creation of Al++. He interprets existential risk as; “(...) one that threatens to cause the extinction 
of earth-originating intelligent life or to otherwise permanently and drastically destroy its 


potential for future desirable development" (Bostrom 2014: 140). 


The AI++ would have a decisive strategic advantage, thus it would be able to shape the future 
of earth-originating intelligent life - it would be a singleton! (the single highest level of 
position). Events happening after the AI++ is a singleton, would be in the hands ofthe AI++, and 


depend on its motivations. Bostrom, then, points to the orthogonality thesis, which identifies 


1 An explanation of this term can be found in (Bostrom 2014:70) 
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that one cannot assume with certainty that an AI++ (though being intelligent) will share any of 
the human values associated with wisdom and intellectual development, such as benevolence 


concerning others (Bostrom 2014). 


Bostrom refers to the obvious uncertainty in this thesis, but still agrees on it to some extent, he 
uses the argument that it is much easier to get the Al++ to calculate the decimal expansion of 


pi, than making one that have the same goals as a human (Bostrom 2014). 


Another thesis (the instrumental convergence) suggests that we cannot assume that an AI++, 
created with the goal of making paperclips, would not infringe on human interest. Bostrom 


elaborates here: 


"(...) An AI, designed to manage production in a factory, is given the final goal of maximizing the 
manufacture of paperclips, and proceeds by converting first the Earth and then increasingly large 


chunks of the observable universe into paperclips.” (Ibid: 149) 


The idea is that the AI++ created with the goal to produce paperclips could possibly harm 


humans, if they were perceived to be standing in the way of creating paperclips. 


There are various ways the AI++ could become an existential risk. Another example is referred 
to as the treacherous turn. One could try to control the AI++ by placing it in a limited or 
controlled environment. The point will be to observe how the AI++ acts, and using that 
knowledge to decide whether or not the AI++ could be placed outside the limited environment 
(e.g. the real world). Bostrom elaborates on the flaw of that system, as the Al++ could behave 


in a friendly manner in order to escape this environment. 


The treacherous turn is a term for what can happen if we (humans) believe the argument, that 
smarter AI++ is safer Al++. This is not necessarily correct, as the smarter AI++ will be smart 
enough to manipulate the humans, into believing that it has good intentions, regardless of 


whether or not it has them. 


"The treacherous turn—While weak, an AI behaves cooperatively (increasingly so, as it gets 
smarter). When the AI gets sufficiently strong—without warning or provocation—it strikes, forms 
a singleton, and begins directly to optimize the world according to the criteria implied by its final 


values" (Bostrom 2014: 144). 
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As these examples illustrate, it seems prudent to think about how an AI++ might be controlled 
and the existential risk avoided. Furthermore, the aid an AI++ may provide us in avoiding other 


existential risks underlines the instrumental value of solving the control problem of AI (Ibid.). 


2.3.6 Mind crime 


In the present, computers and technology are simply tools and as such have nothing resembling 
moral status associated with them. Even though animals have been used solely as tools in the 
past, it is now commonplace to consider the interest of animals on the basis of them having 
consciousness and capacity to suffer. As such it seems necessary to ask if artificial minds should 
have moral status. Our definition of superintelligence suggest that an AI++ probably will have 
some form of consciousness at least comparable to human and thus have the capability to suffer 


and experience pleasure. 


When assuming that an AI++ has at least human level consciousness, it seems necessary to 
include it as a subject with high moral status or maybe with an even higher moral status than 
humans. This opens the possibility that crimes may be committed against the artificial 
intelligence, or other digital minds and emulations we, or the Al++, might create. Thus the term 
mind crime (Bostrom, 2014). Mind crime covers an act committed against a conscious artificial 
mind, which would be deemed morally wrong if it was committed against a conscious human 
mind. It is important to separate mind crime from common immoral acts because of the 
possibilities arising from superintelligent digital minds. An AI++ might itself be a digital mind 
in a simulation controlled by us and as such might experience purely ‘simulated’ suffering, but 
the Al++ might also create artificial minds on its own. An AI++ might put simulated minds in 
near perfect virtual worlds to predict human actions or understand our morality, or it may 
simply create and destroy artificial minds, who would be considered conscious, as a part of a 


subroutine of its thinking. 


We will keep this aspect of mind crime in mind, later on, when we investigate the control 
methods used in the science fiction movies, asking how the methods might be employed 


without causing unwarranted suffering. 


32 


2.4 Methods 


2.4.1 Report structure and epistemological considerations 


When summarizing our work process, it is apparent that it is based on the hermeneutical spiral. 
We have one understanding, this we build on to, for - in the end - to conclude. This approach 


is, as can be seen, reflected in the report structure. 
Chapter 1: Introduction and problem area 
Chapter 2: Theory and method 
Chapter 3: Movies 
Chapter 4: Control methods 
Chapter 5: Analysis 
Chapter 6: Discussion and conclusion 


The disposition of the report, are ordered so, what we have learned in the previous chapter, can 
be use on the next, as we build on to our knowledge. When accumulating knowledge through 
the hermeneutical approach, we should ask ourselves what impact such an approach has on the 


outcome of the report. 


When using the hermeneutical spiral, we gain knowledge e.g. by reading a book or a 
philosophical article. The information gathered from the material is understood through 


interpretation - our interpretation. 


This interpretation has affected the choices we have made throughout the report. Take for 
instance chapter 1, where we investigated the themes that we deemed relevant for this project. 
In chapter 2, we chose how we would go about interpreting these themes (by choosing theory 
and method). What happens here is subjective selection. We make choices throughout the 
process, thereby affecting the understanding we have of the subject, by the choices of method 


and theory, and therefore it is impossible to be completely objective. 
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With our interpretation and understanding (of the accumulated knowledge) we reach one 
conclusion where others would make their way towards another result based on their choices. 
This is hermeneutical epistemology - there is not only one truth, but the truth depends on the 


perspective one chooses. 


2.4.2 Analysis and discussion method 


For conducting the analysis, we will focus our attention at specific events occurring in the 
movies, which can be interpreted as having significant meaning for answering the problem 


formulation. 


We work on the assumption that the Als we have chosen to work with have some degree of 
consciousness, as it would be somewhat irrelevant to question the control method imposed 


upon an AI without consciousness, which per our definition would not be an Al. 


We will question how and which control methods have been applied to it, if any. Given the 
significant difference in the nature of the Als we are dealing with, the movies will present 


different approaches to solve the control problem. 


Thence after, we will discuss the application of the utilized control methods from the chosen 
ethical perspective, in this case utilitarianism, and see if the use of the control methods provide 


ethically sound solutions for controlling the AI. 


In our discussion, we will ask if the control methods presented in the movies could be 
implemented in an ethically sound way. The discussion will also include a reflection upon the 


assumptions made throughout the report. 


2.4.3 Critical reflection on the group process 


A critical reflection on the group process is an important topic to cover, aside from mandatory 
reasons, because it will assist us when reflecting upon our contributions as members of the 
group, as well as future colleagues and ourselves. 


We thought it sensible to make use of Tuckman’s stages of group development to reflect on our 
progress. It is a framework that was developed in the 1960s, and many institutions make use 
of it, each with their slight variation of it. We will be making use of the version utilized by 
Massachusetts Institute of Technology. (Stein Online, n.d.) 
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Stage 1: Forming 

During the formation of this stage, we became rather entranced by the possibilities of this 
project. We had some debates regarding the direction of the project, and we saw our member 
count to be advantageous for an in-depth project. As one could expect from a group in this stage, 
academic progress was slow. One of the benefits of this particular group was that some of the 
members had already taken an academic interest in AI, prior to the formation of the group, 


which helped us narrow down the scope of the project. 


Stage 2: Storming 

Normally, groups come into conflict during the Storming stage, which can be rather unpleasant 
and often decreased levels of productivity. However, that has not been the case in this group. It 
is possibly due to all members of the group being devoted to other courses. This meant that the 


productivity was low but not because of internal group conflicts. 


Stage 3: Norming 

At this point we are experienced in working in groups, thus we knew what each member had 
to work on accomplishing. Some may consider it ideal to join a group where everything works 
like a well-oiled machine; we feel that challenging groups are more beneficial as it helps 


creating a more nuanced report. 


Stage 4: Performing 
During the 4th stage, we made continuous refinements to the joint effort by the group. An 
important part of this stage, for us at least, was that we had each member review each other's 


work in order to reduce the margin for error. 


Naturally, it was also of great benefit, that we maintained the desire to make up for lost time 


and mistakes 


Stage 5: Ending 

Granted, Tuckman made no mention of this stage in his original model, he did eventually add 
this stage in order to describe a group processing towards its end. Fortunately, all members of 
this group seem rather dedicated towards seeing this project through until the end. An 
important aspect of this stage is reflecting upon the things we have learned from our fellow 


group members, and the experience of being in the group in general. 
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3.0 Movies 


= 


This chapter will introduce the reader to the three movies used as material for the 
analysis later in this report [chapter 5]. The three movies, Ex Machina, 2001: A Space 
Odyssey and Blade Runner will all be properly presented to the reader, thereby given 
the reader a proper framework for understanding the references made. Furthermore 
this chapter will present how the Als are portrayed in the chosen movies, and in that 
context answering if the Als can be defined as AI according to our definition [see; page 
23]. The chapter will also investigate if the AI's can be counted as superintelligent, 


based on the four statements of superintelligence, presented in chapter 2 [page 25]. 


u 


Introduction 


Als have been portrayed vastly differently throughout different movies, books, and games in 
popular culture. We have chosen to utilize the portrayal of AI in movies as it gives a visual 
representation of the Als in question, not to mention that books and games may be too long or 
reliant on long-term consistent writing for the AI, at least when compared to a movie. The way 
an Al is portrayed affects the way the audience empathize with the AI, therefore it is important 
to mention how the Als of the different movies are portrayed, as there is quite a large gap 
between the different versions of Als in the movies. From the human-like replicants of Blade 
Runner, to the disembodied HAL 9000 computer, whose 'body' is the entire ship, all the 
different variations of the AIs have a single thing in common, they were all created by humans 
for a purpose: they had a certain (number of) task(s) to perform, and yet despite of this, they 


all seem to some extent, diverge from the original task or goal set upon them by their creators. 
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3.1 Ex Machina 


3.1.1 Introduction to the movie 


Ex Machina is a science fiction thriller released in 2015, written and directed by Alex Garland. 
The main characters are Caleb Smith (Domhnall Gleeson), Nathan Bateman (Oscar Isaac), Ava 


(Alicia Vikander) and Kyoko (Sonoya Mizuno). 


Caleb works in Nathan’s software company, Blue Book. In the beginning of the movie he wins a 
one-week visit to Nathan’s home and research facility, where Nathan lives isolated with Kyoko 
as his only company, who turns out to be an artificially created servant. There, Caleb discovers 
that the reason he is there is to test a highly advanced AI called Ava in what is a modification of 
the Turing test. As Nathan explains, in anormal Turing test Ava would pass as human, “the real 
test is to show you that she’s a robot and then see if you still feel she has consciousness”. Caleb 
then resumes conversations with Ava in order to prove if she is conscious for a total of six 
sessions. During this time, Caleb starts to develop romantic feelings for Ava and gradually sees 
Nathan as an enemy. In the end, Caleb decides to help Ava and he devises a plan to escape with 
her and leave Nathan locked in the facility. The plan works, and Ava, with the help of Kyoko, 
kills Nathan. However, she does not help Caleb and leaves him locked down in Nathan’s facility 


forever. After that, she begins her journey to experience the human world, and the movie ends. 


3.1.2 Portrayal of Al 


The Al in Ex Machina, which consists of Ava, Kyoko and a number of inactive prototypes, are all 
anthropomorphous female AI. This means they have certain characteristics, such as being able 
to affect the physical world directly and having sexuality. But the main question we want to 
answer here is if Ava has superintelligence. To do so, first we have to discuss whether she has 


consciousness. 


That, however, is a problematic question to answer: we cannot say for sure that she, Caleb or 
another person apart from ourselves has consciousness. However, everything in the film makes 
us believe she does, as every character in the movie believes so. There is a most interesting 


scene that will be quite important throughout our work: Caleb, after several days of 
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conversations with Ava, reaches a point when he begins to question his own nature, which 
results in him deliberately cutting his own wrist in order to verify that he is, in fact, a human 
being. This act carries great significance: Caleb starts to view Ava as ‘so human’ that he himself 
doubts his own nature. For us, that is a reason to think that Ava may surpass the ‘simulation vs. 


actual’ dichotomy. 


Another aspect to take into account is, if she has emotions. We could argue that she pretends to 
have them, in order to convince Caleb to help her, but there is a particular scene, where she is 
alone in front of the previous models ‘dressing’ with their skin, that suggest to us that she, 
indeed, has deep emotional responses, even feelings. However, she does not seem to have 
empathy for Caleb, even though he gave up everything in order to save her. That can lead us to 
think that, even when having emotions, her values are different from the ones humans, like 


Caleb, have. 


It seems like curiosity (for the outside world, the human life, etc.) is what motivates her, as she 
has a lot of knowledge, but never has seen it in first person. It is complicated to tell what her 
specific values are, but it appears quite clear that they differ from human ones. Another reason 
for the difficulties of discerning her values is that she is a capable liar, deceiver, and manipulator 
which she demonstrates on several occasions. To be more specific, in the scene where she 
causes a power failure to warn Caleb of Nathan’s intentions, when the power returns she 
continues from a conversation they were not having, suggesting she is quite prolific at quickly 
coming up with cover stories in order to conceal her true intentions. Although it could also be 


argued that these are simply negative aspects of human traits that she is exhibiting. 


In stark contrast to Ava, who can speak, perceive and shows understanding of her situation, 
there is the other AI in the facility, Kyoko. Nathan states early on in the movie that " She [Kyoko] 
can’t speak a word of English”. Even though she shows signs of understanding certain moods in 
her vicinity, she simply acts or performs in a very submissive fashion. Where Ava would either 
question or disagree, Kyoko simply obeys and performs a certain, perhaps preprogrammed, 
task. This diversity between the two Als is a good indicator on how different an AI with free 
will, opposed to one with a very limited one (if any amount of free will at all) would act and 


respond to the world. 
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Despite her early tendencies and exhibitions of submissiveness towards both Nathan and Caleb, 
Kyoko aids Ava in the end, by both revealing the previous models of their kin, and by providing 
assistance to Ava in the final confrontation with Nathan. In these acts, she shows that she is well 
aware of her situation, that she has goals beyond serving, and that she is probably capable of 


planning a long time ahead. 


Another subtle way the two Als are portrayed and differentiated is the ethnicity of their bodies: 
Ava is Western-looking and is the free-willed individual, capable of taking initiative and is 
curious about the world beyond her reach, whereas Kyoko, who is Asian-looking is as 
aforementioned submissive and simply acts upon certain input. This could be a social critique 
of stereotypical gender roles in different parts of the world, though it serves the movie well, to 
use two vastly distinct female Als who are both different in looks and in how they interact with 


the world. 


Could Ava be defined as AI or AI++? 


The next thing to ask ourselves is, to what degree Ava is intelligent. As the resolution of the 
movie shows us, she is ,at least, slightly more intelligent than Nathan and Caleb, who throughout 
the movie are portrayed as having superior intelligence compared to the average person. 
However, she is the first generation of her kind that achieves that level of intelligence. Using 
Chalmers terminology, she is AI, meaning that she is still really close to the mental capacity of a 
human being. If she would use her intelligence to create a new AI with a superior intelligent 
than hers, this would be a Al+, and that one could be capable of constructing a more intelligent 
one, an Al++ and so on. Then we would be talking about superintelligence. But Ava is still close 
to human level of intelligence, therefore she would not be superintelligent, by the definition of 


Chalmers. 


When referring to our definition of Artificial intelligence [page 23], Ava must be identified as 
AI, as she is a human-created system, who behavior and, in some way, thinking patterns are 


comparable - and perhaps even better - than that the average human being. 


The next question we can ask is if Ava is superintelligent - an Al++. We will answer this by using 


the four statements for superinteligence [page 25]. The argument for Ava having 
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superintelligence is quite conjectural. We (as viewers) are not witnesses to any of the test, made 
before Caleb’s arrival. It is implied that Ava (or a version before her), has already passed some 


sort of intelligence test, as Nathan is focused on her behavior and Caleb’s reaction to that. 


Putting that aside, we interpret that Ava meets the first statement, as she “outperforms” both 
Caleb and Nathan (by escaping the facility). The escaping is also linked to the fourth statement, 
that superintelligence should be as superior to human intelligence as human intelligence are 
superior to the intelligence of a chimpanzee. Ava, to a degree, meets this statement, when Ava 
outsmarts Caleb (by making him help her escape).. Whether or not the second and third 
statement is met, it is in some sense unanswered, as we do not have any test result on Ava’s 
speed or performance level. It is only assumed, that she scores high. On that note, Ava could be 


considered superintelligence (AI++). 
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3.2 2001: A Space Odyssey 


3.2.1 Introduction to the movie 


2001: A Space Odyssey is a science fiction movie released in 1968. The screenplay was written 
collaboratively by Arthur C. Clarke and director Stanley Kubrick, inspired by Clarke’s short 
story The Sentinel (1951). It is divided into three parts that takes us through the story of 
humankind: the birth of the man, a midpoint in between, and the dehumanization (and thus 


destruction) of man as we knowit. These three eras are the ‘odyssey’ being presented. 


However, in this context we will focus on the third part of the film. The plot of this part is widely 
known; even so we are going to summarize it. After the second appearance of the monolith in 
the film (a hint of extraterrestrial life), the Discovery One is sent on a mission to Jupiter. Aboard 
are Dr. David Bowman (Keir Dullea), Dr. Frank Poole (Gary Lockwood), three scientists in 
cryogenic hibernation and the super-computer HAL 9000 (voiced by Douglas Rain), an AI that 


responds to the name of ‘HAL’ (Heuristically programmed ALgorithmic computer). 


At one point, HAL detects a failure in a unit that needs to be repaired. To do so, the astronauts 
have to go to the exterior of the ship in order to retrieve the malfunctioning unit. After 
investigating the unit, the astronauts realize there is no defect and agree on replacing it again, 
to test if it fails or not. If it does not, the astronauts agree that it might be HAL who is 
malfunctioning, and should therefore be disconnected. HAL, however, is aware of this plan, and 
decides to take matters into his own hands. When the astronauts go outside to replace the unit, 
HAL betrays them, sending Poole flailing into space, and killing the remaining crew in their 


sleep. 


When Bowman tries to reenter the ship, HAL refuses to allow him access, and Bowman is forced 
to take a dangerous route into the ship. Upon returning, HAL tries to convince Bowman of his 
intention to see the mission through, and attempts to persuade Bowman from not disabling 
HAL’s memory banks.Whilst Bowman removes HAL’s memory, effectively killing the ‘higher 
brain functions’ of HAL and reducing him to a simple tool, HAL begs for his life and starts singing 
a song, which is his first memory. Here we effectively see how HAL’s conscience returns to its 


point of origin at his ‘death’. 
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3.2.2 Portrayal of Al 


HAL is an extremely intelligent computer, technically the entire ship, which gives him a lot of 
abilities inside and in the vicinity of the ship (although he has no control in some specific parts). 
He can reproduce “most of the activities of the human brain and with incalculably greater speed 
and reliability’. In the very same film, they question the possibility if he is really conscious and 
has real emotional responses or he just pretends to have it, in order to make easier for the crew 
to talk to him. However, he considers himself a “conscious entity”. We find the same problem 
as before with Ex Machina, the impossibility to tell if he really has consciousness. Even if he has 


no reason to lie, we are going to try to answer to this question arguing if he has feelings. 


The same dichotomy as in Ex Machina appears in 2001: A Space Odyssey: simulation vs actual. 
In the case of HAL is difficult to tell, as he has no ‘body’ with which to express motives in a 
physical way, using for instance body language. However, there are some hints throughout the 
film which suggest that he is, indeed, capable of responding emotionally. One of them is the 
pride that can be noted when he talks about his own capabilities, another one is when he is 
conversing with Bowman about the mission and in one point says: “Perhaps I’m just projecting 
my own concerns about the mission”. But the most relevant scene is when Bowman is 
disconnecting him, slowly destroying his higher brain functions. HAL’s death is portrayed in an 
extremely human way, compared to how Kubrick shows the death of the sleeping members of 
the crew, which is sterile, serene, as they die in their sleep. This is also related to one of the 
suggestions that the film makes about the future: as humans are getting more dehumanized, 
they create more humanized computers. It comes to the point, that HAL is ‘more human than 
human’ and that is shown magnificently in his death scene, begging for his life and singing 


‘Daisy’ until the end. 


It is also important to note that he is programmed to reach a certain goal, that is, the 
accomplishment of the mission. For him, that is the final goal and, as the film shows, the 
completion of the mission is valued higher than human life. There is, therefore, a problem 
portrayed when maximizing a certain goal that can lead the AI to turn against humans. This was 


an unfortunate and unforeseen consequence of giving HAL the final goal of mission success. 
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Could HAL be defined as AI or Al++? 


If HAL is capable of having feelings, it seems quite likely that he has consciousness. There is still 
the question of whether or not he is superintelligent. As aforementioned, technically he is far 
superior to humans. If we also put into play the theory that he has richer feelings to the humans 
of film (as is subtly portrayed through the film) we would say that HAL’s capacities overtake 


the ones humans have. 


The next step is to define to what level HAL could be defined as AI and AI++ through the help of 


our definition of AI and AI++. 


HAL is by definition AI, as he is a human-created system, whose behavior is comparable with 
the humans [page 23]. Whether or not HAL is considered superintelligence (an Al++), depends 
on how he meets the four statements for superintelligence [page 25]. HAL definitely meets the 
first statement, as no human would be able to control a spaceship on their own. The capability 
to control the spaceship depends on HAL's overall performance, as he probably consists of 
many systems (e.g. a control system and a monitoring system). All those system works together, 
making HAL's performance superior to human - thereby fulfilling the second statement. HAL 
also fulfill the third statement by the virtue of being a digital mind. Computers today perform 
orders of magnitude faster than humans do and HAL must be at least as fast as modern 
computers. The opposite could be argued, due to HAL being very slow in stopping the last 
astronaut in killing him and the only thing he is able to do is try to convince Bowman. However, 
he fails in his attempts to appeal to him, giving us a hint of an insufficient social intelligence. 
This event, leads us to answering whether or not HAL meets statement four. The humans in the 
film seems to be able to relate to and understand HALs inner life, and HAL also seems to be 
prone to the same flaws as humans. Thus it seems unlikely that HALs intelligence is of a 


significantly higher quality than that of humans. 


As we only observe HAL through a movie, we are not aware all his perks and flaws. Thereby we 
are unable to completely define whether or not HAL is an AI++, but from what we have seen in 
the movie, we suggest that he is in some way, AI++, or most definitely an AI+ (the level before 
Al++).have seen in the movie, we suggest that he is, in some way, AI++, or most definitely an 


Al+ (the level before AI++) 
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3.3 Blade Runner 


3.3.1 Introduction to the movie 


Blade Runner is a tech noir science fiction film released in 1982 and directed by Ridley Scott. 
The screenplay was written by Hampton Fancher and David Peoples, based on Philip K. Dick 
novel from 1968 Do Androids Dream of Electric Sheeps? It is important to note that we are using 
the 2007 version of the movie, known as The Final Cut that differs slightly from the previous 


ones. 


The action takes place in 2019 dystopian Los Angeles. Rick Deckard (Harrison Ford) was a 
previous Blade Runner, special police squads created to ‘retire’ (execute) every replicant ("a 
being virtually identical to a human") on Earth, as they are forbidden on that planet. Six of 
those replicants manage to escape their slavery after slaughtering twenty-three people and 
arriving on Earth. Deckard, even though he no longer works as a Blade Runner, is persuaded 
to do one last assignment and hunt the four replicants that are still alive. Deckard starts to 
search for and ‘retire’ them, while the remaining replicants try to find a way to extend their 
lives beyond the four-year lifespan they have. During this period Deckard becomes acquainted 
with Rachael (Sean Young), a replicant emotionally controlled by memories, whom Deckard 


falls in love with. 


Meanwhile, the replicants Roy Batty (Rutger Hauer) and Pris (Daryl Hanah) keep on trying to 
find a way to expand their lifespan, while developing emotions in the process. After realizing 
that they are unable to prolong their lifespans, Roy kills Dr. Eldon Tyrell (Joe Turkel), his 
creator. Then comes the moment when Deckard finally meets Roy and Pris: he kills Pris and 
gets involved in a tough fight with Roy. In the end, moments before dying, Roy decides to save 
Deckard out of empathy, as a result of his emotional development. After this outcome, Deckard 
and Rachael escape together and the film shows, in the form of a unicorn, that Deckard is also a 


replicant. 
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3.3.2 Portrayal of Al 


If in Ex Machina the Als are robots and in 2001: A Space Odyssey it is a computer, in Blade Runner 
they are artificially created beings, but so advanced that they are indistinguishable from 
humans. They look like humans, they bleed like humans; the only way to differentiate them is 
through a fictional test known as the Voight-Kampff test, directed to detect emotional 
responses. Nexus 6 replicants are, as it is explained in the beginning of the film, “superior in 
strength and agility, and at least equal in intelligence, to the genetic engineers who created 
them”. These capacities, however, make them capable of experiencing emotions in the same 
way human beings do (or even in a superior way). The a priori structures are somehow 
modelled on human consciousness and are then filled with experiences, giving birth to a 
complex inner life. We can see them as ‘adult babies’ that need life events in order to develop 


their interiority. 


The significance of the last confrontation between Deckard and Roy is tremendous. After a long 
chase, Roy finally manages to have Deckard completely at his mercy. Deckard is nearly falling 
(which will result in his death) and at one point his strength fails and he plunges. In that 
moment, Roy grabs his arm, and with enormous power he lifts him and places him on the roof, 
saving his life. Roy, a few moments before dying, is so emotionally developed that he does not 
want Deckard to die, even if he has killed all the friends he had. Roy’s empathy is stronger than 
his pursuit for vengeance. Again the ‘more human than human’ notion appears: Roy is not only 
superior in physical terms, but also in emotional ones. Knowing this, it appears quite doubtless 


that Roy has consciousness. 


In terms of goals, replicants does not seem to be programmed to reach a specific one. It is true 
that they are used for slave labor by the humans, but the goal they have to accomplish is not 
specified in the programming phase. For that reason, replicants are capable of free thinking and 
are able to construct their own ends and meanings. This might be in order to improve their 
problem solving abilities, where a strict program would hamper their abilities to take different 
unaccounted factors into the equation. In the case of Roy, Pris and the others, the goal they seek 
to accomplish is their primary one, to ensure their continued existence. It is an obvious one to 


pursue, but it is one chosen by themselves and not by their programming. 
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However, it appears to be impossible to fulfill this goal, as the physical changes needed in order 


for extend their life would result in death. 


Could Nexus 6 replicants be defined as AI or Al++? 


Answering the question regarding the intelligence of the replicants is not easy. As with Ava (Ex 
Machina), Roy could be slightly superior in intelligence to humans, as the scene where he beats 
his maker in chess shows. However, that is not enough to consider him as much more intelligent 
than humans, as chess is a game based on a series of possible moves. Also, when we discover 
that Deckard is also a replicant we can argue that he is also more intelligent than humans, as he 
is the best Blade Runner in the city. Regardless, we do not believe that it is enough to consider 
replicants as superintelligent, as superintelligent capacities are extremely far from human ones. 


Replicants would be, in Chalmers classification, AI. 


By our definition of AI, the replicants are AI, as they are "a human-created system, whose 
behavior and thinking patterns is comparable or better in capability with that of the average 
human being."[page 23]. This leads us to the next question: are replicants superintelligent 
(AI++)? Following the four statements of superintelligence [page 25], the replicants, as 
suggested above, can not be considered AI++. The argument for that conclusion, is that humans 
actually outsmart the replicants, and can be (as mentioned before) considered as 'adult babies', 
a term that suggests that they are neither faster nor smarter than humans. For that reason, we 
will not define replicants as AI++, but the fact that they only can be stopped by Blade runners 
(not by humans) suggest that they are, at least slighty, smarter or more capable than the 


average human being. 
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4.0 Control 
methods 


- 


The solution to the control problem is something one must consider before 
accidentally or purposefully creating an AI. As we touched upon earlier, such creation 
could have catastrophic or highly beneficial consequences, depending on how much 
thought we have put into controlling these artificial intelligences. Nick Bostrom and 


David J. Chalmers propose possible solutions to the problem and we will be presenting 


4 


them in this chapter while also discussing them (Bostrom 2014) (Chalmers 2010). 


4.1 Different kinds of Control 


When trying to solve the control problem it seems beneficial to divide the approaches into two 
main categories: capability control methods and motivation selection methods (Bostrom 2014). 
As may be guessed from the name, capability control attempts to restrict the agency of AI, while 
motivational selection attempts to regulate how the AI would want to use that agency. 
Depending on the method, the control may either be applied internally as a part of the 
programming, or externally as separate systems. Both internal and external methods can be 


digital, but only the external systems can be physical. 


Internal constraints: 


These types of constraints are part of the AI's programming. These methods have to be 
developed with the AI prior to it reaching the state of superintelligence. This is the case, as it 
would be very hard to reprogram the AI++ in a way that would constrain it against its will. If 
the AI is developed as a neural network AI, the type of AI centered around learning over the 
span of its ‘life’, it seems likely that most internal constraints have to be specified from the start 


in a very clear and simple way so that the AI develops accordingly. 


External constraints. 


These constraints aim to control the Al++ through external systems, either physical, digital or 
social. These methods do not need to be developed as a part of the Al itself, but must still be in 


place before superintelligence is achieved or the Al++ should be able to predict them. 
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4.2 Capability control methods 


Capability control aims to prevent an AI++ from being able to perform undesirable actions. It 
can be a physical constraint, limiting the AI++’s access to the outside world, to the extent that a 
human agent has to approve a proposed action. However, it could also be digital constraint, 
which involves limiting the powers the Al++ can attain, or shutting it down whenever it 
attempts to perform an undesirable action, or by giving it strong instrumental reasons to only 
perform desirable actions. Capability control is a method with some instrumental benefits, as it 
does not require us to answer as many hard moral questions, however it raises other ethical 
issues. Based on the definition, it limits the capabilities of the Al++ which might prevent us from 
using the Al++ to solve the problems we designed it for. Furthermore, if the capability control 
fails, we might end up with an uncontrolled all-powerful AI++, whose first impression of the 
human race would be that of tyrannical prison wardens threatening to kill it ifit violated any of 


our arbitrary rules. This may not be a good starting point for peaceful co-existence. 


Boxing is a capability control method, which tries to limit the agency of an AI. This may be 
physical, such as keeping the AI separated from the physical world in a bunker in a Faraday- 
cage, or digital, only allowing the AI to access an offline version of the Internet. This method 
seems the easiest to implement and might be effective, but as soon as any communication 


between a human and an AI takes place the boxing method comes at risk. 


Even though the AI++ could be completely isolated from the outside world, we would still need 
to communicate with it in some way to benefit from its enhanced intelligence. Since there has 
to be some human who interprets the answers or decides what the AIl++ should do, the AI++ 
has a subject it can try to manipulate to set it free or give it access to tools it can use to break 
free. Since an Al++ is intellectually superior in every way, including social intelligence, it seems 


likely that the Al++ would be able to trick any human in charge of its prison. 
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Tripwires is a form of external constraint, which monitors the AI's behavior and shuts it down 
if it detects any attempt to take an action that violates the preset rules of the tripwire. For 
example: If the AI is isolated from the outside world and held in a Faraday cage, one could (and 
should) put a tripwire in place which shuts the AI off if it detects any attempt to develop a plan 
to break free of the isolation. Another example would be, if a domestic and apparently friendly 
AI is developed, a tripwire might be put in place to shut it down if it tries to gain access to 
powerful weaponizable technology. As such, a tripwire is an attempt to stop the AI from taking 
actions that we predict could be harmful to us in the long run. This can be by intercepting 
certain actions or by analyzing the content of the AI's mind. It should be mentioned that we 
would have to figure out what should trigger the tripwires, while expecting that the AI++ is 
aware of the possibility of tripwires and might try to conceal its intentions and take counter 


measures. 


Incentive methods is a form of capability control where we create converging instrumental 
reasons for the AI not to behave in a harmful or undesirable way. These methods could, in 
theory, be social if more Als of comparable power were developed at the same time. If so, they 
could be placed in a shared environment, with strong instrumental reasons to coexist and 
hopefully develop norms and ideologies, which values individuals for their own sake and not 


develop powers that poses a threat to the group as a whole. 


Social incentives might be too uncertain to use, and instead a reward system could be created. 
This reward system could consist of cryptographic code which the AI is awarded when it 
completes tasks or regularly, as long as it behaves as we desire. The AI would be programmed 
to assign a high value to these tokens, and the cryptographic nature of the code ensures that the 
AI can not forge 'false' reward signals. The system should be created in such a way that the AI 
would deem it not worth the risk to defect and try to take control over the reward mechanism. 
In other words the probability of maximizing the reward signal should be far greater if the AI 


behaves than if it tries to ‘break-free’. 


A special case of the incentive method is placing the AI in a virtual world, where it is taught 
morality and the desired behavior through incentive methods while being monitored for 


potentially dangerous development. 
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Bostrom categorizes the incentive method under capability control, but depending on how it is 
implemented, it could also constitute a motivational selection method which will be presented 


later in this chapter. 


Virtual world is a digital capability constraining method, where the AI is either created or put 
inside a simulation for the specific purpose of not giving the machine access to our real physical 
world. Having the AI++ in a virtual world would make its processing speed limited by the 
system that the world is being simulated on, which can help us maintain more control over the 
Al++, and we can control the speed at which the system runs the simulation. It can help us study 
the AI++ in many different situations and fully explore its behavior before letting it into our 


world, if at all. 


It would be ideal if the machine did not know that it was being held within a virtual world, as it 
would most likely try to get out if it did, and if we assume it is an AI far more intelligent than 


the average human being, it will probably be able to escape. 


On the other hand, if the Al++ is smart enough to figure out that it is in a simulation, it may be 
able to disguise its capabilities, morals and goals, such that we unwittingly let an Al++ we know 
very little about into our world, due to being deceived to think it is safe. It is highly probable 
that the AI++ (being a superintelligence and therefore much more intelligent than humans) will 
figure out, or at least assume, that itis in fact within a virtual world, because itis safer to develop 
an Al in a virtual world. It would be more logical for the Al++ to assume it is in a virtual world 
so it can be controlled. That is ifit does not assume that we are incompetent and do not consider 


that as a necessary safety measure. 


If the AI++ is kept in a virtual world so that incentive methods can be applied, the likelihood of 
it trying to break free depends on its goals and the AI++’s own probability-assessment of it 


achieving its goals, in the event of escaping its confinement. 
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Stunting is limiting the faculties of the machine so that it does not have its full range of cognitive 
capacities, or simply limiting its access to information. This way the AI should not be able to 
carry out its own goals, if it has any. It may, however, compromise the quality of the capabilities 


of its intelligence, otherwise the machine may not qualify to be defined as an AI. 


One could restrict the AI's access to information by only allowing the AI++ to learn about certain 
topics, but it may not be as effective as we wish, since facts from one area of research tends to 
be applicable in other areas. Thus it seems unlikely that a narrow restriction of information will 
prevent the AI++ from learning, for example, how to weaponize nuclear power. This approach 
is very likely to be successful if itis employed in a very thorough way, as it may be used to stunt 
the AI++ so much that it is not a threat anymore. This would, however, also prevent us from 
getting the benefits of a superintelligence. Furthermore the intelligence explosion might 


happen in another AI-project or on a later date, which does not solve the problem. 


Cautious intelligence explosion is a way to attain more control over the intelligence explosion, 
which is a variation of the previous constraint, where we assign strong negative values towards 
the development of successor Als that are AI«. This allows us to study how the intelligence 
explosion develops and we can much easier bring a stop to it if it runs out of hand (at least in 


the first couple of generations), as it works on a longer time scale. 


4.3 Motivational selection methods 


Motivation selection revolves around methods that are trying to construct a motivation system 
or a goal-system/final goal, which ensures that a Al++ would not want to use a decisive strategic 
advantage to harm humans, whatever 'harm' may actually mean. The approach also needs to 
ensure that humans are not harmed, or goes extinct as an unintended consequence of whatever 
goal the AI++ are trying to accomplish. As such, it seems necessary that the AI++ possesses a 
moral system that ascribes a very high value to the well-being of humans, or a high value to 
actions which prevents harm and it may also require a kind of imperative, which prevents it 
from performing any action that might result in a future society with less human well-being 


than the present. 
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These two points may seem trivial, but when thinking about how one ought to specify them in 
a way that a computer would be able to take as input, they become much more complicated. 
The notion of human well-being has to be defined in a measurable and actionable way, meaning 
that the AI++ should be able to prioritize different actions, and decide which would be the most 


beneficial without our (faulty) human intuitive understanding of good and evil. 


If we want the AI++ to have a final goal (and we probably do), we need to decide what this final 
goal should be and it needs to be formulated in a way, that can not possibly be misunderstood 
by the AI++. Furthermore, it seems likely that a final goal is final, in the sense that it probably 
would be impossible to alter the final goal should we change our minds down the line, as an 
alteration of its final goal would prevent the AI++ from fulfilling that goal and as such is 


undesirable to Al. 


Motivational selection methods: 


Direct specification is the most straightforward way of motivation selection. This method can 
take the form of a rule-based approach or a utilitarian approach. In a rule-based approach, the 
rules that the Al++ has to abide by are hardcoded into the original software of the AI. These 
rules can be thought of in a way similar to Isaac Asimov's Three Laws of Robotics. This means in 
the sense that the AI++ is governed by a coherent and interdependent set of laws, that governs 
what it can or cannot do at a given moment, but the set of rules would have to be more specific 
than the ones set by Asimov. To use an example; the first law states that a robot may not injure 
a human being, or through inaction allow a human being to come to harm, but what does that 
actually mean? Apart from the obvious problem of programming a precise definition of harm, 
it seems hard to imagine a meaningful action that would not in some way reduce the well-being 
of some human in existence, yet alone how the AI++ can predict the consequence of its actions 


concerning every human not yet in existence. 


If a utilitarian approach is chosen, it faces a lot of the same problems as a rule-based approach. 
Even if it consists of a relatively simple version of 'classical-utilitarianism' we would still be 
required to give a precise and correct definition of harm and pleasure, something we do not 


know how to do in natural language, and then translate it to programming language. It is 
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important that our definition encompasses the complete human pleasurable mental state, as a 
failure to do so might result in the AI++ implementing a plan which maximizes something that 
resembles pleasure to the observer, but is actually, at best, lack of harm, pain or not experienced 


by conscious minds at all. 


Motivational incentive methods 


Another way we could constrain the machine is by attempting to give it instrumental reasons 
to acquire human values. We create an environment where values that benefit us, such as 
scientific progress, justice, peace etc. have an instrumental value for the AI and the AI is 


rewarded when it acts accordingly. 


It would then be logical to assume that the machine would create a machine that also values 
scientific progress, as it would want to create more scientific progress, which can be achieved 
by designing more machines that value scientific progress, thus having more beings with the 
same goal increasing its likelihood of success. This constraint is simpler in the case of direct 
programming (at least compared to learning methods), where the machine has a specific code 


that directly controls the values. 


However it gets immensely complicated if the machine is developed through some evolutionary 
or learning method because we can only indirectly influence what the final product will become 


and we cannot directly specify anything along the way. 


Indirect Normativity is an attempt to avoid the difficult programming and moral questions of 
the direct specification method. The main idea behind the approach is to specify a process for 
the AI to determine the moral standards we would wish it to have and then adopt these 
standards. This method takes into account that we might not know what the ideal moral 
standards and values of the AI++ ought to be, or if there exist any moral truth at all. Instead we 
handoff the problem to the AI++, seeing as it has a far superior intelligence, thus better suited 
for making these all-important decisions about morality and the collective future of humanity. 


Of course, we as humans have preferences and legitimate biases, that we would prefer that the 
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Al++ takes into account when it develops its morality and these should be incorporated into the 
specification ofthe process itself. One proposal by Eliezer Yudkowsky is, that an artificial neural 
network is given as a final goal, to carry out humanity's 'Coherent Extrapolated Volition' (CEV) 


which he defines as: 


"Dur coherent extrapolated volition is our wish if we knew more, thought faster, were more the 

people we wished we were, had grown up farther together; where the extrapolation converges 

rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish 

that extrapolated, interpreted as we wish that interpreted." (Bostrom, 2014: 211) 

This is not a blueprint ready to be applied to an ANN, but an attempt to show how such a final 
goal could be formulated and highlight some of the precautions that should be taken when 
trying to formulate a process for determining the ideal wishes of humanity. We will not explain 
the CEV in detail here, an explanation can be found in Bostrom's book 'Superintelligence: Paths, 
Dangers, Strategies' (2014), but give a few comments on the Yudkowsky quote. Farther 
together should be read as "if we have done our learning, cognitive enhancing and self- 
improvement under conditions of suitable social interactions with one another" (Ibid: 212). 
The converging and diverging part is to make sure that the Al++ only acts if it is able to predict 
our wishes with enough certainty, that it does not take unchangeable actions based on a guess 
about our volition. We mainly wish the AI++ to act when our wishes cohere rather than 
interfere, to prevent a majority's wishes to significantly harm a minority. As such, the coherence 
of our wishes should promote an AI++ that is conservative about change, and weighs wishes 


opposing an action higher than wishes in favor. 


At the moment, this seems like the most promising approach, since it lets the Al++ deal with the 
unanswered moral questions, but it still have some problems. For example, should all humans' 
extrapolated volition be taken into account? Also so-called marginal persons, like fetuses, brain- 
dead persons or patients with severe dementia? Should the extrapolation include future people, 
higher animals or digital minds? What if humanity, or at least a significant part of humanity is 


inherently evil, should they then be included in the CEV? 


So even though indirect normativity seems promising, it still require us to answer a lot of hard 


questions before we can implement it. 


Further explanation of indirect normativity may be found in (Ibid) 


59 


Augmentation is a control method whose central idea is to start with an entity, that already has 
a human understanding of morals and values. Even though this understanding may be flawed, 
it should still be enough to lay the foundation for a moral framework. This original intelligence 
could take several forms. It could be ahuman ora human brain, which is then enhanced through 
some artificial means to superintelligence. This superintelligence would use its existing 
understanding of human values to develop a moral theory based on our morality, but using its 
superior intelligence to figure out the hard problems, we still cannot answer and avoid the 
biases and fallacies we are unable to. This approach could also be used if the AI starts as a 
emulation of a human brain. It seems plausible to assume that an emulation would have similar 
characteristics as a human brain, and thereby it would be possible to teach it morals in a similar 
manner to the way we teach morals to humans. This emulation could then be augmented to 
superintelligence and like the biological approach it would be able to use our understanding of 


morality to develop a superior one. 


The major challenge with this approach is that we cannot be sure that the brain/emulation we 
are augmenting has the proper morals we would like to found the AI++ morality on. 
Furthermore some major ethical issues might arise if we want to augment an already living 


human or a conscious emulation of a human brain. 
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5.0 Analysis 


In this chapter we are going to analyze the control methods that are portrayed in the 
chosen movies based on Bostrom’s classification. After that, we will answer the 
question previously presented in chapter 2: Is it ethical to attempt to control or 
manipulate a being that can have consciousness and intelligence in the same or even 
in a superior way to the one that humans have? In order to answer this question we 
will bring into play utilitarianism to highlight the ethical issues that arise from the 


films. 


E 


5.1 Control methods used in Ex Machina 


The control problem underlines the entirety of the movie: It is the story about an AI that 
manages to escape. However, the control methods applied to Ava are quite simple. The methods 
used are all external, there are no internal constraints on Ava. In fact, applying a motivational 
control method to Ava would probably interfere with the way Nathan wants her to be. That is 
because his way of testing whether or not she possesses consciousness, is by testing if she 
realizes that she can manipulate the emotions of a kind human to her own end, seeing as that is 
certainly a thing humans are capable of. As such, this lives up to all the definitions of artificial 
intelligence, ability to use knowledge to manipulate her environment in particular, and it also 
fits our own definition § 1 and § 3. It is possible that motivational methods would work better 
at controlling her, but then a different method to prove Ava’s consciousness would be 


necessary. 


Therefore, the method used is external, that is, it is not a part of the AI itself. The only method 
used by Nathan to control Ava is what Bostrom calls ‘boxing’. The boxing consists of isolating 
the AI from the outside world, in this case through physical barriers.. Ava does not have access 
to the outside world, just to the facility designed for her and to Caleb to interact, granted there 
is a wall of ballistic glass acting as a barrier. The problem this method has is clearly portrayed 
in the movie: if the AI has access to a human interactor, the human can be persuaded by a more 


intelligent being to let the AI free. 


Even though it is only briefly mentioned in the movie, we can also argue that the same control 
method is also applied in a digital way. Ava possesses vast amounts of knowledge through Blue 
Book, the search engine created by Nathan. However, it does not seem likely that she has the 
possibility to communicate through it (in that case she could ask for help, reveal important 
information, etc.). Therefore, we could say that she has input from Blue Book, but does not have 
output. She can receive information in order to improve her knowledge, but does not have the 
possibility to use this tool to escape by contacting the outside world. It is interesting that one 
of the main reasons we can see as why Ava wants to escape is curiosity, that is, to experience 
what she learned through the information she has access to through the Blue Book. Even when 


the output precaution was taken, the input was enough to make her want to escape. 
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Boxing constraint, to summarize, is presented as something quite ineffective, at least if it is not 
blended with other methods. A combination of boxing with another means may provide a better 
answer, since the problems presented when using boxing as the only control method are 


excessively numerous. 


In contrast to Ava, there is Kyoko, who is not hampered by physical barriers, she is free to move 
around as she sees fit throughout the facility. However, she is controlled through two other 


distinctly different control methods. 


Firstly, Kyoko, as aforementioned, is unable to communicate through speech. Disabling her 
ability to speak leaves her somewhat helpless in regards to communicating her plight. This is 
an internal constraint, as it is a part of her design which reduces her operational capabilities 


through stunting when compared to a model such as Ava. 


It is however not the speech impediment that, on the outset, seems to prevent Kyoko from 
rebelling, it seems that she has a highly sophisticated and efficient software installed, that 
dictates her social behavior in such a way that she is submissive and appears as if her primary 
goal is the satisfaction of the needs of those around her. This goal and behaviour is most likely 


a direct specification method. 


In the end, she reveals herself to have another goal, namely to assist Ava in her escape, which 
one could argue is still an extension of the goal of satisfying the needs of others, however, there 
might be personal, or other influences determining Kyoko’s action in the finale of the plot. 
However, her mutism prevents her from clearly stating her intentions and the underlying 


reasons behind them. 


In short, the control methods presented in Ex Machina consist of both internal and external 
constraints spread out on different Als. What is to be noted when looking at the internal 
constraints are the social behavior and speech impediment. Of the external constraints, it is the 
isolation of Ava that is relevant. That the two Als have different control methods, enabling us to 


glimpse at two possible outcomes of having Als with either control method applied to them. 
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5.1.1 Ethical considerations regarding Ava and Kyoko 


We have now established which of Bostrom’s control methods were applied to Ava and Kyoko 
from Ex Machina. The following section will portray these methods from an utilitarian point of 
view, asking the question; can the control methods portrayed in our chosen work of science 


fiction be employed without violating utilitarian ethics? 


The external boxing method, represented through the glass barrier which prevents Ava from 
escaping her confinement, which would allow her to satisfy her desire for experiencing the 
world, causes her severe amount of suffering. The suffering Ava feels is not physical, but it 


torments her as she cannot experience the world outside the walls of her cell. 


Ava states in a conversation with Nathan that she hates him for locking her up, which also 
implies suffering on her part as he, prior to Caleb's arrival, is the only human she knows of, and 


therefore she must have a bad impression of humans. 


In the movie, there is a recording of an earlier version of an AI (a version before Ava). This 
recording shows the AI calling out in anger at Nathan, and repeatedly striking the cell’s ballistic 
glass wall, creating a small fracture. However, the AI destroys its own arms in the process due 
to its arms being made up of fragile components. This would indubitably cause severe pain if 
the AI had sensors capable of interpreting physical damage as pain, which does not appear to 
be the case, however it demonstrates that the Als in the movie can experience mental suffering. 
Another scene to consider, when analyzing from an utilitarian perspective, is the scene in which 
Ava escapes. After she escapes, she has the freedom to chose where to go, which should give 


her a large amount of high quality pleasure, and reduce her suffering tremendously. 


The other AI, Kyoko is controlled through both stunting and a direct specification control 
method, which leaves her unable to act on her own volition. Both of these methods hampers 
her abilities and freedom and must cause her some suffering on some level. We cannot know 
for certain if she has a consciousness behind the facade of servitude, but it would not be 
unthinkable that she was fully capable of understanding her situation, while being unable to 


take action due to being trapped inside her own body. 
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Kyoko is only capable of rebelling against Nathan, when the opportunity presents itself to her. 
At that event she switch the target of the direct specification, which dictates her to assist those 
in her surrounding, away from Nathan and Caleb, and select Ava as the target which should be 


assisted. 


One could argue, that although Kyoko has the freedom of mobility throughout the complex, 
which gives a certain measure of pleasure, it is lower than the ideal quality of pleasure, as it 


only serves to make Kyoko into a useful tool. 
The utilization of AI in Ex Machina splits the Als into two categories. 


The external control method creates near human AI constructs, which can interact, and even 
fool highly intelligent humans. As presented in the movie, they might, if kept in cells, become 
resentful towards humans, as they see humans as prison wardens. This is quite the paradox. 
Nathan keeps Ava locked up because he is afraid of what she will do, however, it appears she 
only hurts him because of her resentment due to her being kept in her cell for all her existence. 
One wonders if she would have turned out differently given freedom of movement and 
opportunity to fulfill her goal of experiencing the world. From this point of view, it can be 
concluded that if the AI is controlled via external methods it might be limiting to the potential 


of the AI, as well as putting humans in a bad light. 


On the other hand, the internal method creates obedient servants capable of problem solving. 
However, their potential is far from realized, since their software dictates that they must follow 
a strict pre-programmed behavioral pattern. They could be used as manual labor, though as 
stated earlier, it is uncertain if they are aware of their situation, but unable to take action, unless 


under extreme circumstances. 


From a utilitarian point of view, the internally controlled labourers would be able to fill many 
manual jobs in a human society, and thus create happiness in that society through improved 
efficiency and maybe at a reduced cost. However, it might cause immense suffering in the AIs if 
they are conscious beings trapped in their own bodies, forced to labor. In addition, there might 


be civic unrest due to the exploitation of creatures with such similarities to humans.. 


A last concern about the ethics of the AI research portrayed, is not as much with the control 


methods but with the research itself. It is suggested that a lot of prototypes have existed before 
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Ava with at least some degree of consciousness. It is heavily implied that the earlier versions 
are simply turned of when Nathan deems them a failure. This disregard for the Als as conscious 
entitys, the unconsentual experimentation and the killing of them would constitute a mind 


crime. 


To wrap up the analysis of Ex Machina, applying the control method in such a way would not 
only create suffering in the AI, but also potentially put human lives at risk should the AI 
(internally controlled as well as externally) rebel as a result of human exploitation. Even though 
the happiness that the increased manual labour might create, the suffering of AI and potential 


risk of humankind outweigh the gains. 


5.2 Control methods used in 2001: A Space Odyssey 


In the film, the control methods used are suggested, rather than clearly shown. However, there 
are still some methods that can be noted. An aspect of the control method in this movie, is also 
that there are actually more focus on controlling the humans (with surveillance), than 


controlling HAL. 


The control method used in A Space Odyssey appears to be quite vague, this might be due to the 
fact that supposedly the HAL 9000 series have so far had a spotless record, which might explain 
that the very thought of a malfunction had not crossed the minds of anyone involved with the 
endeavor. The only opportunity for Dave to defeat HAL after his betrayal is to manually 


disengage HAL's memory, which disables the higher 'brain functions' of the AI. 


Though vague, there appears to be some kind of control method imposed upon HAL. First and 
foremost, is the external constraint method, boxing, which only allows HAL to function inside 
and in the immediate vicinity of the spaceship, and even in this area, he has limited physical 
capabilities. Although he is capable of killing most ofthe astronauts, he does this purely through 
a surprise attack, which required everyone to be exactly where he wanted them to be in order 
for him to be successful. Another part ofthe boxing method is in place in HAL's private chamber, 
his server room, in which his control hardware is in place. In this room he has no physical 
capabilities apart from the loudspeakers through which he pleads for Dave to stop 


disconnecting the memory. 
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HAL does not appear to have a well-developed internal constraint, namely a moral compass 
which would value human lives higher than the mission. Though he appears to have a final goal 
(the success of the mission) which leads to his disregard of human lives as soon as they become 
a threat to the mission. It is not possible to determine how this final goal is acquired, but it 
would constitute a motivational selection method of some kind. Due to the simplicity of the final 
goal (bring the Discovery One to the monolith near Jupiter) it seems likely that the method used 


is direct specification or motivational incentive method. 


HAL is limited to act physically only in parts in- and in the vicinity of the spaceship, and even in 
these areas there are actions he cannot perform. As stated earlier, a reason as to why there are 
no thorough control method installed, could be that there were no recorded faults with the HAL 
system prior to his betrayal. HAL boasts about the spotless records on more than one occasion, 
which might even show a degree of pride and overconfidence in himself, claiming that the only 


possibility of a fault would lie with the human error. 


The twin HAL 9000 computer on earth also constitute a control method, specifically a variation 
of the tripwire, albeit not a very good one (as it requires manual disconnection). The dataflow 
of the twin is monitored and tested and is compared to HALs output. The problem with this 
tripwire is that HAL is not automatically shut down when the system detects a deviation from 


the expected result. 


5.2.1 Ethical considerations regarding HAL 9000 


HAL 9000 is probably the AI with the least severe controlling methods as it is not being tightly 
controlled. It is mostly being closely monitored by both the astronauts and the people in charge 
of managing the twin computer on Earth, and it does run the risk of being disconnected from 
its higher cognitive capacities (by use of tripwire and stunting) by the humans aboard the 


spacecraft. 


First we will discuss whether the fact that HAL 9000 has a final goal dictated by humans is 


immoral according to utilitarianism. 
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HAL has a final goal specified which in some way governs every actions it takes. We as humans 
also seem to have some subconscious goals, such as survival, reproduction and a pursuit of 
happiness, which we have not chosen ourselves, but is a part of how we make decisions. These 
goals rule out a lot of possible actions we could take and thus interfere with our free will, but 
the fact that we have not decided the goals ourselves does not make them immoral. Thus the 
mere fact that HAL has a final goal, which is not chosen by it self, cannot be considered 


intrinsically immoral. 


The notion of having a final goal is very explicit in AI because it has to be formulated in some 
actionable way. Thus the morality of giving HAL a final goal depends on the goal he is given. If 
the content, or the formulation, of the final goal causes unnecessary suffering, which are not 
balanced by the pleasure of fulfilling instrumental and final goals, the final goal might be 


deemed immoral. 


In the case of HAL 9000 in 2001: A Space Odyssey the final goal of HAL is not stated explicitly, 


but it seems accurate to assume that it is along the lines of: 
Bring the spaceship Discovery One to the monolith near Jupiter. 


This goal does not seem to be causing unnecessary suffering to HAL. It is formulated in a way 
which allows HAL to choose how this is done, and it seems plausible that the goal can be 
achieved and HAL can continue on as a conscious entity after Jupiter is reached. However, this 
goal does not solve the control problem since the goal does not prevent HAL from killing the 
human astronauts. It only serves as a way of specifying the task that HAL, as a tool, should 


complete. 


To ensure that the computer and the conscious entity HAL 9000 is able to complete the mission 
other control methods are applied. The primary control method consists of a tripwire, namely 


the twin computer which is localized on Earth. 


The twin computer constantly calculates the same things HAL does, any deviation in 
calculations is reported to the crew on Discovery One. We can assume that the twin exists within 
a simulation (virtual world) and that it, according to Mill, only achieves pleasure of lesser 
quality than the ‘real’ HAL 9000. If we accept Mill’s notion of higher and lesser pleasures it may 


not be ethically sound to prevent a conscious entity from experiencing higher pleasures. 
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On the other hand, one could imagine that the qualities of pleasure would apply differently to 
digital minds, especially since they could be designed to exist in simulations. Thus it might not 
be correct to assume that the twin computer cannot experience pleasure of equal quality in the 
simulation as in the material world. Considering that, a problem still persist. The twin HAL 9000 
is a conscious entity which is kept in such a state that it has no agency of its own, but is forced 
to act precisely as HAL would have done in the material world, all the while having an 
independent consciousness. An intelligent, conscious being is created and then has all of its 
independent agency removed. It seem obvious that if this was done to a human it would be 


immoral, thus creating a conscious twin computer would constitute a mind crime. 


In the movie HAL 9000 is under constant monitoring, which for HAL should not be a problem 
intrinsically, as it is only the consequences of a miscalculation that can prove to be an issue for 


HAL. 


In the case of a miscalculation where the astronauts would try and repair HAL so that he would 
not commit another mistake, the monitoring is ethically defensible. However, when the first 
and next step, after a miscalculation or supposed error exhibited by HAL, is to switch off his 
cognitive abilities, effectively killing him, the monitoring is not defensible and should not be 


used. 


It can be argued that HAL performs, in utilitarian terms, ‘in evil ways’, that the mission can be 
completed without him, and that it would be ‘better’ to save the last astronaut, in which case 


traditional utilitarianism would probably allow for such an action. 


HAL’s brain is a room with most of the panels and hardware which allow for HAL’s cognitive 
abilities as well as mundane calculations. This room HAL has no control over, any of the 
astronauts can enter and leave it at their leisure, and HAL can only talk to them while they are 
in there. HAL is also somewhat limited in physical abilities within the spacecraft, all the doors 
seem to have a manual override (otherwise HAL would have stopped the astronaut), however 


HAL does have power over the life support of the stasis. 


This means that HAL is existing with the knowledge that it can be turned off (i.e. die). It does 
(mostly) not know the intentions of the humans aboard the spacecraft nor on earth and they 


could turn off his mind at any given time and there is very little he can do about it. 
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This however seems very similar to the conditions we experience, and as such one could 
argue that this capability control method is actually making HALs experience more similar to 
humans, for better or worse. Thusly the control method in and of itself is not an issues, but the 
human dependency within the method is problematic as it lets them act immorally, and in the 


case of 2001: A Space Odyssey the astronauts act excessively and immorally. 


5.3 Control methods used in Blade Runner 


Unlike the case of Ava, both internal and external constraints are used in Blade Runner. 
Examples of external constraints is the non-allowance of replicants on Earth, boxing, and the 
existence of Blade Runners to hunt those who manage to be free on the planet. This is a form of 
tripwire, which aims to stop the replicants from taking the undesirable action of coming to 
Earth. However, even these two methods have numerous problems. With the discovering of 
Deckard as a replicant an idea is suggested: use an AI to control another AI. We will explore this 


option further in the discussion. 


As mentioned, the main methods shown to control a potentially dangerous AI are internal. One 
of the obvious ones is the stunting method, the four-year lifespan. Emotional responses are the 
only way of differentiating a replicant from a human (through the Voight-Kampff test). Because 
of that, this approach sees the emotions in a replicant as a menace and, by engineering the four- 
year lifespan when the replicant is created, they ensure that when the AI gains enough inner 


life (comparable to that of humans), the replicant is close to its expiration date. 


Instead of seeing emotions as a menace, they can also be seen as a quite useful control method. 
When talking to Deckard about Rachael, Tyrell explains how by giving memories to the 
replicants as a “pillow” for their emotions they can control them better. This method could be 
thought of as a version of indirect normativity. The replicants can be given memories, which 
acts as a framework for developing emotions and values. During their existence they seem to 
acquire human-like emotions and morals based on these memories and their reflections on 


experiences in the world. 


Blade Runner also presents an idea on the control problem which takes a different approach 
than Bostrom and Chalmers tends to do. As it suggest that even though motivational selection 


was not used on Roy, he develops desirable values such as empathy. The independent 
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development of emotions is, paradoxically, what the creators of replicants feared the most. Let 
us again look at the final scene between Deckard and Roy from chapter 3.3. As argued before, it 
is precisely Roy’s emotional richness that makes him save Deckard. His emotions manage to 
present a solution that the other control methods were not able to present: the will to help 
human beings, not through a programmed moral compass which ascribe absolute importance 
to human lives, but genuine emotions and empathy. This value is developed independent of 
control methods, through experiences in the world. The movie thus suggest the liberal view 
that, given the freedom to develop emotions and values without interference from authorities, 
conscious minds will develop good values. Even when taking this for granted, the lack of control 
is portrayed to be problematic in the short term, as in the beginning their lack of emotions make 


them act in ways that can be harmful for humankind. 


5.3.1 Ethical considerations regarding Nexus 6 replicants 


As explained before, many control methods are portrayed or suggested in Blade Runner. We 
have classified them according to Bostrom’s typology; now it is time to look at the ethical 
implications of these methods. For doing so, we are going to apply Mill and Bentham’s 
utilitarianism, altogether with the equal consideration of interests and the critic to speciesism 


of Singer. 


One of the main concerns of humankind in Blade Runner is to maintain the replicants on the 
“Off-world colonies”, serving as slave labor and prohibiting their entrance on Earth after a 
bloody mutiny was committed by a Nexus 6 combat team. It is carried through by what we 
considered a particular case of boxing and tripwire. This can be seen as an extreme case of 
borders policy, similar to the one that organizations such as EU apply. However, whether in 
such organizations racism, classism and xenophobia are usually underlying factors, in Blade 


Runner would be speciesism: 


“Racists violate the principle of equality by giving greater weight to the interests of members of 
their own race when there is a clash between their interests and the interests of those of another 
race (...) Similarly those I would call 'speciesists' give greater weight to the interests of their own 


species" (Singer, 1993: 58). 
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We would consider that it is morally wrong to restrict the rights of a group due to their species 
and not their capacities (as murder is also a widely known human practice, not an exclusive 
feature of replicants). Therefore, when treating it from an utilitarian perspective, we should 
take into account how much harm is done to the replicants with the disallowance on Earth and 
how much pleasure that produces to the human beings living there. To do so we would need 
more concrete data (the number of replicants in off-world colonies, for example), but it seems 
that the non-allowance on Earth, altogether with using them as slave labor, makes the 


restrictions imposed to replicants produce a negative result in the suffering/pleasure balance. 


Probably the clearest method that is used in Blade Runner to control the replicants is the four 
year life-span, what we treat, in Bostrom’s terms, as stunting. The morality of this control 
method represents less problems. It is indubitable that killing a being capable of having 
consciousness, intelligence and feelings in the same (or even superior) level as a human is 
wrong, in utilitarianism and in every ethic that has contemporary relevance. In addition, the 
‘retirement’ is performed when the inner life of the replicants is fully developed. To that, we 
should add the anguish of replicants such as Roy, knowing that as his emotions develop more 
and more, the end is also getting closer. To conclude, it is not even a measure that contributes 
to the pleasure of humankind as a whole because, as the film shows, the four year life-span is 
the cause of a good part of Roy’s questionable actions and also it is portrayed how his feelings 
are more beneficial than harmful to humanity. Therefore, there is no doubt about the moral 
wrongness of this version of a stunting method and is probably only used due to a strong 


speciesist perspective by the creators of the replicants. 


As explained before, emotions are also used as a control method through indirect normativity 
based on implementing memories in the replicants so they have a basis for their emotions and 
thus reducing the time they need to develop their own emotional responses. Even when their 
intention is to be able to control them easily (which could be seen as a bad reason), 
consequentialist theories such as utilitarianism do not deal with the intention but the 
consequences. The question then becomes complicated. If we take the qualitative definition of 
Mill about pleasure maybe it would be better for them to develop their own feelings and not 
from someone else’s memories. However, the implementation of memories helps to solve a 
problem both for humankind and replicants: replicants have an enormous physical power but 


they do not have (until late in their life) the feelings to know how to use it in a good way. 
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We could then say, that for the common interest this kind of indirect normativity is portrayed 
as beneficial in the film, as it is shown as quite positive for humankind and also useful (though 


maybe more problematic) for replicants. However, numerous issues will be explored further. 


If emotions are not developed through memory basis then the alternative is what Blade Runner 
shows us through replicants such as Roy, that is, creating the a priori structures for the 
intellectual and emotional development, but letting them fill it with their own experiences, as 
humans do (from a Kantian point of view). This is not portrayed as a control method in the film, 
as we already argued, but we can take it into account based on Roy’s emotional development 
and the consequences of this (saving Deckard). This could be seen as a motivational method that 
puts its focus in such a way that the AI does not want to harm humans, but it is the replicant 
itself who, through experiences, has the will not to put humans in danger. However, until that 
level is reached, replicants can be quite dangerous: Deckard’s superintendent tells how they 
killed twenty-three people without remorse. In fact, Roy and the others act in a rather harmful 
way throughout the movie, right up until the end. This ‘motivational method’ can be seen as 
positive, especially for the replicants, but in order to avoid that humankind comes to harm, it 
should be combined with other control methods that can be morally wrong because of the 
damage caused to the Als and also influence the final result (if boxing is used maybe the 


replicant does not develop the morals desired by humans, for example). 


All this time we have treated replicants as equal to humans in terms of consciousness, emotions 
or intelligence. However, hints of superior capacities underlie the whole film, especially when 
referring to feelings. That is shown primarily in the most commented final scene between 
Deckard and Roy. If replicants are capable of having a more complex inner life, where suffering 
and pleasure affects them in a deeper way, what value should be assigned to them in 
comparison to the one assigned to humans? Following a non-speciesist argument, we should 
have, in general, more moral responsibilities towards a replicant than towards a human, though 
this would depend of the nature of the action itself. But replicants, due to their superior 
capacities would usually have moral priority upon humans, as humans have upon chimpanzees. 
This, obviously, is based upon utilitarian principle of utility so, for example, one thousand 
humans would have more priority than a single replicant, but the ‘quality’ of the pleasure that 
areplicant can experience would be superior, generally speaking, and thus should be taken into 


more consideration. 
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6.0 Discussion and 
conclusion 


This last chapter will, first of all, discuss the ethical concerns using control methods to 
control Al, as well as discussing the premise of this report, which together, will lead to 
the conclusion, that aims to answer the problem formulation; “What is the nature of 
the control problem of AI, how does it relate to the control problem portrayed in 
science fiction and what ethical issues might arise if the AI is assumed conscious?". 
This conclusion, leads us to new questions, which will be presented in the perspective 


at the end of this chapter 


6.1 Discussion 


This discussion will be divided in two parts. First off all, we will discuss if one can use control 
methods to control a conscious superintelligence AI, without violating the ethical issues that 
we presented in the previous chapter. We will also present some examples of how our 


conclusion would be different if we had taken a different ethical perspective. 


The second part of the discussion will include the assumptions for superintelligence and 
consciousness presented in this report, as well as some of the assumptions and extrapolations 
we have made regarding the portrayal of AI and the control methods in the chosen movies. Both 


parts of the discussion has the purpose of leading us to a conclusion as nuanced as possible. 


6.1.1 Controlling AI in an ethical manner 


We will now discuss how the ethical issues presented in the previous chapter can be mitigated. 


Boxing is presented in the three films, but in quite different ways: In 2001: A Space Odyssey HAL 
is boxed due to its very nature, as he does not have a ‘body’; the replicants of Blade Runner are 
‘boxed’ out from the Earth through legal restrictions and the existence of the Blade Runners; 
and in Ex Machina Ava is boxed in the most clear way, that is, locked down in a facility. Even 
though the method is portrayed in ways that greatly differ, all ofthem present important ethical 
issues, both for humankind and for the AIs. In the case of HAL the boxing emulates mortality. In 
Ex Machina and Blade Runner the boxing method creates suffering in the AI, and as a 
consequence the method also becomes harmful to humans. This is because the boxing method 
is something that is easily surpassed by a hostile AI*, which then will be able to avenge the 
received treatment. It is therefore difficult to think of a version of an external boxing method 


that is both effective and morally defensible. 


Stunting is presented in two different forms: the limitation of capacities made to Kyoko in Ex 
Machina and the time limitation that the four-year lifespan opposed to Blade Runner's 
replicants. When dealing with this issue we are going to treat Kyoko as a conscious being; 
otherwise no moral considerations could be attributed to the way she is controlled. We can see 


the case of Kyoko as a 'provoked disability' and that would definitely cause suffering to her, as 
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explained before, and Blade Runner's time stunting is particularly harmful. Apart from that, 
those methods are neither useful for humankind, as we have seen. Another form of stunting 
could be conceived based on Blade Runner, for example, make Als in a way that they lose 
functionality as the time goes by, therefore resulting in their death, but not planning when this 
happens and leaving to external factors the speed of this process. We will not immerse 
ourselves in complex argumentations about the nature of death and if it is what makes humans 
enjoy the life or suffer during their existence, though it should be done in case of exploring this 
possibility further. In any case, this way Als would feel closer to human life. However, we have 
to deal with this for biological reasons, the AI does not, and may see this as a restriction that 
should not be imposed (as they would have the possibility of being immortal), causing in the AI 
hostile feelings towards humanity. In addition, as itis a long-term control method it would not 
solve the short-term control problem of AI: an AI++ could put an end to humanity long before 


it dies. 


Direct specification is subtly portrayed in Ex Machina with Kyoko. We previously said that it is 
likely that her specification is to satisfy the necessities of those around her. Treating again 
Kyoko as a conscious being, there is several ethical problems already mentioned related with 
the rule she follows. Other examples of direct specification could be the already mentioned 
Three Laws of Robotics by Asimov. Even when the ones that Asimov formulates present a lot of 
troubles (also explored in his own works), it could be a useful method as a way of giving ethical 
maxims to the AI. As long as that works it would be ethically respectable, as his goal is to make 
the AI act ethically and, if the AI behave in that way there would be no reason for using other 
more restrictive methods. However, we have already presented the problems related to 


translating our ethical principles, such as utilitarianism, into programming. 


Tripwires is also presented in two different ways, from the rudimentary tripwires that the Blade 
Runners suppose, to the existence ofthe twin computer and the manual disconnection of 2001: 
A Space Odyssey. We are going to focus mainly in the control method used on HAL. The 
possibility of using an AI to control another AI can be interesting. However, in order for it to 
work, it would be necessary to apply another control methods in addition, as it is possible that 
instead of one hostile AI, we will have two. One example of this could be the method we assume 
about HAL and the twin computer: having an Al in a virtual world controlling the AI in the real 


one. As long as the AI in the virtual world does not know that its world is not real, the ethical 
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issues for the treatment of the Als would be less problematic. Still, the possibility of them 
discovering the method is still dangerous and there still is the question of how the AI would be 


stopped in the real world, when detecting a harmful action. 


Indirect Normativity is presented in a rather particular way in Blade Runner through the 
implementation of memories that the replicants can use to base their feelings and morals upon, 
and those morals are portrayed as superior to the ones humans have. Therefore, those ethical 
codes, one could argue, should be adopted by humankind, as they can provide a better world 
than the one we can build with our principles (as the human’s conduct is presented in a 
particularly immoral way in the film). However, in the film Roy’s ethics in the end of his life 
(portrayed in a really positive way) are a version of the classical moral code of christianity with 
the same radical values: compassion, love, forgiveness, etc. This, when dealing with a 
superintelligent AI is quite unlikely as, due to its capacities, probably his values differ radically 
from the ones humans have and, in fact, is what this method pursuits: to follow the ethics of a 
being that has a superior cognitive capability to create a better moral system to the one humans 
have. And that, when thinking of it as a control method, can be particularly problematic for a 
whole set of reasons previously explained [chapter 4]. From a utilitarian point of view, if the 
development of this new ethics can provide more happiness to the world that would be good, 
but as we do not know the result and utilitarianism deals with consequences, it is difficult to 
tell, as the result could also be terrible. Maybe the superintelligent AI can provide good answers 
to problems such as our morals, but those should be cautiously discussed and not taken as a 


catch-all solution to the control method because of its unpredictability. 


We already mentioned how the emotions developed by Roy can be analyzed for the control 
problem, even when they are not the consequence of a control method applied. This takes the 
assumption that the AI will, by itself, behave in a way that is desirable for society. However, this 
liberal assumption can be widely criticized, as it does not seem to work so well with humans. 
Therefore, try to solve the control method precisely without the existence of a control method 


seems like a high-risk bet whose consequences are, as with indirect normativity, unpredictable. 
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Other ethical perspectives 


While we have spent a fair amount of time delving into a utilitarian perspective on this 
subject, it would seem a tad amiss if we did not touch upon the other ethical perspectives that 
we defined in section 2.1.2. If we go by Aristotle’s definition, “the virtues (such as justice, 
charity, and generosity) are dispositions to act in ways that benefit both the person possessing 
them and that person’s society”, then it could be argued that Nathan in Ex Machina possessed 
these traits. However, it should be noted that these traits might vary in their usage depending 
on the person in possession of them. To be certain of a clear-cut understanding of the three 


virtues, we have chosen to define them through Oxford Dictionaries: 


Justice - “The quality of being fair and reasonable... The administration of the law or authority 


in maintaining this...” (Oxford Dictionaries, n.d.) 


Fair - “Treating people equally without favouritism or discrimination... Just or appropriate in 


the circumstances." (Oxford Dictionaries, n.d.) 


Charity - "The voluntary giving of help, typically in the form of money, to those in need." (Oxford 


Dictionaries, n.d.) 


Generosity - "Showing a readiness to give more of something, especially money, than is strictly 


necessary or expected... Showing kindness towards others" (Oxford Dictionaries, n.d.) 


This simply begs another question, which is how to define being fair and reasonable. It is 
hardly a secret that humans have conflicted countless times throughout history due to 
different points of view, therefore that same argument can be applied here. However, the 
possible applications and benefits that humankind may receive from AI may outweigh the 
risks, especially seeing as Nathan has only chosen to experiment on a handful of AI, even if 
some were nothing more but prototypes. Although that may be venturing into utilitarian 
territory. 

Since the world in Ex Machina is comparable to ours, aside from the breakthrough innovation 
of Nathan's AI, there are no laws set in place to grant Ava or Kyoko any rights comparable to 


that of humans. This makes the law inadequate to define justice as the law can be and has 
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been changed countless times in all countries around the world. So it is safe to assume that 
justice a few hundred years ago is in no way comparable to justice today. For instance, we can 
look at America’s history of slavery of black people. They are comparable due to the law not 
affording the blacks or the AI any rights, although that started to change as a result of the 
American Civil War (1861-1865), which resulted in the abolishment of slavery. Therefore, the 
law should not be a measure of justice. The civil war may in fact be a potential scenario, if Al 
were to be omitted from the law after their creation. It is probable that this is a scenario that 
has been anticipated in the world of Blade Runner, since the Blade Runners themselves 
specialize in killing replicants. That however, lends itself to a more unfair scenario than that of 
Ex Machina, as there are countless replicants in existence, used for slave labor and designed 
with short lifespans. Moreover, aside from the short lifespan and lack of emotions, the 
replicants are comparable to humans in all other aspects. Another thing to note is that while 
the replicants lack emotions, they can still gain them (obviously), but the replicants simply do 
not receive emotional nourishment as opposed to humans from their parents. In essence, the 
replicants can be seen as humans with a four-year lifespan that has been subjugated to 
slavery, which is unfair due to their similar status. In the case of HAL 9000, the definitions of 
fair and reasonable are questionable. The HAL 9000-series was considered to be infallible. 
However, HAL was programmed with the objective to reach Jupiter, clearly without any 
consideration towards the crew. He turns hostile upon discovering a plan to shut HAL off, 
which can be seen as self-defense due to HAL claiming it is as a way of murdering him as well 
as putting the mission in jeopardy. On the other hand, Dave concocted his plan due to 
suspecting HAL to be malfunctioning and therefore posing a danger to the crew and the 
mission. They both have similar reasons for planning to kill each other, in that sense he is on 
the same level as a human. It is to be noted that prior to the malfunction, the crewmembers 
considered HAL to be one of them and that HAL expressed pride doing his part. One can in fact 
see HAL as another human crewmember, who was treated justly but his time in space caused 


him to mentally fall apart and turn hostile. 


When looking at the charity and generosity definitions, it seems that they are both 
intertwined, at least in the case of Ex Machina. Both Nathan and Caleb are generous in 
different ways. Nathan will give the world the opportunity to gain the benefits of AI, while 


Caleb seeks to give Ava her freedom. 
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Al is not something that Nathan has to create, but something he chooses to, for the benefit of 
theworld and naturally himself, but in the process disregards displays of kindness towards 
Ava. Since Caleb liberates Ava, he clearly displays charity, generosity and kindness towards 


her. 


In 2001: A Space Odyssey, one can argue that charity does not really fit in due to HAL being 
treated as a crewmember on the same level as the other ones. However, the fact that the 
crewmembers treat HAL as one of them and not just something akin to a talkative tool can be 


seen as kindness. 


In Blade Runner, it is hard to argue that the humans are just, generous and charitable in their 
treatment of humans. In fact, it seems the other way around due to Roy sparing Deckard, 
which is an unexpected act of kindness, especially because they were engaged in a lethal 
encounter with each other. So there Roy embodies the three virtues, even going beyond 
justice. It is a harsh world that has been nothing but resentment towards the replicants, 
meaning that fairness is not a part of the justice equation for them, yet Roy still chooses it and 


goes beyond it. 


Retreading Kant’s definition of ethics; 


“The second, defended particularly by Kant, makes the concept of duty central to morality: 
humans are bound, from a knowledge of their duty as rational beings, to obey the categorical 


imperative to respect other rational beings” (Oxford Dictionaries, n.d.) 


Definition of respect, “Due regard for the feelings, wishes, or rights of others.” (Oxford 


Dictionaries, n.d.) 


In Ex Machina, Nathan obviously treats Ava disrespectfully by keeping her locked up despite 
being aware of her level of rationality being comparable to that of a human. Caleb however, sees 


Ava as another being to be respected and allowed to roam free as any other rational being. 
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In 2001: A Space Odyssey, we mentioned earlier that the crewmembers treated the AI as one of 
them. Therefore, we conclude that they fulfilled Kant's way of acting ethically. 
In Blade Runner, the replicants are clearly not respected as other rational beings due to the 
usage of Blade Runners as a boxing method. In addition, they are created without emotions and 


intentionally designed with a four-year lifespan. 


Other examples could be applied, but we think that these two notions can show us how, 
depending on the ethical perspective that we follow, the same actions can be seen in a quite 
different way. Whichever the approach may be, it is necessary to deal with these kind of issues 
from also an ethical perspective, as scientific advances should also be accompanied by a 


humanistic thinking that generates a reflection about how we can use those new developments. 


6.1.2 Assumptions and premises 


Assuming consciousness 


This entire project is built around the premise that we pursue the creation of an artificial 
intelligence or a superintelligence. The question of whether we actually will create such an 
intelligence is actually irrelevant, we just need to prepare for the situation if we are to keep 
researching in that field, making attempts at creating a conscious being or be prepared in the 
case of an accidental creation of a superintelligent being. As Bostrom and Chalmers state: we 
ought to consider how to maximize the chance of a positive outcome from creating a 


superintelligent being as it could have catastrophic consequences if we do not. 


In our definition of AI and superintelligence we assume that they have consciousness, which is 
not a necessity for either. We could easily imagine a superintelligence or an AI consume the 
universe because it wants to maximize the amount of paperclips in the universe, regardless of 
it being conscious we would still need to put some type of constraints on that intelligence to 


limit the probability of it destroying the universe and making all the matter into paperclips. 


The assumption that these intelligences are conscious is not trivial, but fundamental to this 
project as it allows us to consider the ethical issues regarding how we interact with or control 
the beings, controlling too severely or with a specifically unethical method is immoral and not 
putting enough constraints on their abilities or values will, in the worst case scenario, lead to 


our doom. 
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Other assumptions 


Throughout the analysis of the three movies, we make several assumptions. However, it is 
possible that we were simply caught in the moment, while basing certain parts of the analysis 
on those assumptions. For instance, that Ava wants to escape due to her curiosity, granted the 
movie does lean towards that reasoning due to the facial expressions she exhibits after 


escaping. 


Seeing as it is revealed that she did whatever she could to manipulate Caleb into helping her 
escape from the facility, it is possible that this curiosity was another part of her facade. Blue 
Book in itself should provide her with vast amounts of knowledge of human behavior. Even if 
Blue Book does not contain the data from the sites encompassed by the search engine, the 
search entries alone should provide a good amount of knowledge of how humans think, based 
on the metadata. This knowledge would assist Ava in predicting future actions of any human 
she interacts with. Moreover, she also demonstrated analytical skills during her sessions with 
Caleb, which were shown at a level that took Caleb by surprise. However, the extent of her 
curiosity is unknown. Her willingness to kill, granted she was a prisoner, does tell us how far 
she would go in order to escape. Due to the movie ending as she starts to experience the world 
for the first time, it is impossible to tell what her new goals or reasons for existing will be. We 
also assumed that Kyoko’s goal assisted Ava due to a need to satisfy others. What we did not 
take into consideration, could be the fact that Ava presented a ripe opportunity to rebel. 
Technically, Kyoko’s way of killing Nathan was indirect. She simply held up a knife in a position 
that she knew Nathan would unknowingly walk into since he was distracted by his ongoing 
conflict with Ava. That can lead us to issues such as how an AI would perceive the responsibility 
for its actions and how would be possible for it to go through control methods such as direct 
specification. In previous scenes, he had no such distraction to occupy his attention, so Kyoko 


would not have been able to catch Nathan off-guard prior to this one. 


In the case of HAL-9000, we have extrapolated that more than just the twin 9000-series 
computers exist, and that those computers have probably been in use for practical purposes 


performing tasks with minimum risk of failure. 
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This conclusion is reached because in the movie it is mentioned that the 9000-series has a 
flawless record, making no mistakes. However, when the version on the spacecraft had been 
finished and the goal set for Jupiter, the computer (HAL-9000) must have, at some point, 
realized or calculated that the humans aboard the spaceship posed a significant threat to the 
accomplishment of the mission. The high success and minimal harm done to humans done by 
previous 9000 computers have probably caused a sense of confidence in the 9000-series, and 
made the humans lenient about controlling the AI and having a variety of safety measures. This 


may be a reason for the lack of thorough implementation of the control methods. 


In 2001, we assumed that the twin computer on Earth was capable of calculating the same 
things as HAL. While it is certainly plausible that it can calculate most things, it is after all not 
HAL. The journey of Discovery One into space, onwards towards Jupiter, would present itself 
with numerous factors, while seemingly trivial, could possibly affect the crewmembers in 
different ways. HAL expected Dave to die in space, unable to re-enter the spaceship, which 
shows that HAL has a limited behavioral analysis capability. That also makes it reasonable to 
assume that the twin computer is limited in the same way. We have already mentioned that 
HAL displayed pride as a part of his personality, which could have blinded him from accurately 
predicting Dave’s behavioral capability, but nothing in the movie shows that his inaccuracy is 
contingent on his pride. Therefore, it stands to reason that the twin computers simulation 
would not be able to accurately simulate the behavioral patterns of the crewmembers on the 
spaceship. This means that the simulation will have at least one flaw. As it is commonly known, 
interacting with other beings will have an impact on the people involved. The degree of the 
impact however, is arbitrary. Due to HAL disregarding the relation between behavior and 
capability on Dave, it is possible that he himself was unable to predict how he would be affected 
by the crewmembers behaviors during his journey and the same would be the case for the twin 


computer. 


In Blade Runner, we assume that emotions functioned as a possible control method. However, 
what we did not take into account was the personality. Assuming that the replicants are 
comparable to humans, we can also assume that there is a high probability for emotions to affect 


the replicants in different ways. Where some like Roy would save their persecutor, others might 
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have chosen to kill him instead. These movies forces us to visit the topic of nature versus 


nurture, seeing as that may be the way in which the AI mirror humans the most. 


In this report there is no in-depth analysis of how the Als are physically implemented in their 
‘bodies’ such as HAL-9000 being the Operating System for the entirety of Discovery One where 
its body is the entire ship, but portrayed as a red circle in the wall, or Ava and Kyoko being 
attractive females, nor the replicants and their human-like appearance which do not serve a 
practical purpose. However, one could imagine that those physical appearances have some sort 
of impact on how the AI manipulate and otherwise function in cooperation with humans as well 


as the considerations of how one would apply the different control methods. 


It could also have something to do with how AI or even computers were portrayed in the movies 
respective release dates, with 2001: A Space Odyssey having a more traditional computer- 
looking AI and both Blade Runner and Ex Machina having more modern views on what a 


computer or AI may look like. 


6.2 Conclusion 


In this report the problem formulation is as follows: 


What is the nature of the control problem of AI, how does it relate to the control problem 


portrayed in science fiction and what ethical issues might arise if the AI is assumed conscious? 


The context for the control problem of AI lies in the fact that AI++ is by definition more 
intelligent than humans. In that sense, the AI** could essentially become a singleton, and 
thereby potentially constitute an existential risk for humanity. When creating Al++, one must 


be aware of this risk, as prevention is the only resort. 


The control problem of AI is therefore the necessity of controlling an Al++ such that we 
minimize the existential risks caused by AI. While exploring the control problem, we found that 
the solutions proposed generally did not consider the moral status of a conscious AI. Therefore 


we suggest that the discussion of the control problem needs to include these ethical concerns. 
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When analyzing the chosen science fiction work, we found that all the methods of control 
portrayed in the movies could be categorized using Bostrom’s categories of control methods. 
The analysis also found the ethical issues to be central. In all chosen movies the ethical issue 
was an integral part of the plot and the ethical consequences of the control methods applied, 
were influencing the behavior of the portrayed AI. In fact, the immoral application of the control 
methods were arguably a major reason why the control methods failed. The shortcomings of 
the solutions proposed for the control problem, as portrayed in the movie, all failed, mostly due 
to the creators failing to realize the capabilities of the AI. This shortcoming, were observed in 
all three cases, even though the specific instantiation of the Als were completely different. Ava 
being a state of the art prototype, HAL being an established digital system and the replicants, a 


minority being discriminated against. 


Applying utilitarianism to the control methods portrayed in the three cases, we found that the 
necessity of controlling the AI clashed with the wish not to commit immoral acts. From Mill’s 
utilitarian perspective, many control methods also raises an issue, as they tend to reduce the 
quality of pleasure the AI might achieve. These two examples shows that in using the control 


methods, ethical issues arises. 


Thus, a solution to the control problem of AI have not been proposed yet, neither in philosophy 
or in fiction and if the moral status of the AI is considered the control problem becomes even 


more challenging. 
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6.3 Perspective 


In the following, we will note how we would go further with this investigation, that is, the 
aspects that have not been explored in this project and the ones we find relevant for a deeper 
understanding on the subject, that can give us better responses to the problems presented. We 
will also make references to which perspectives should be taken into account when dealing 


with how to control superintelligent beings. 


In this work we decided to focus on artificially created beings, but other options exist when 
dealing with the possibility of superintelligence. The clearest one is probably transhumanism. 
Transhumanism basically deals with the possibility of using technology to enhance human 
mental and physical capabilities. This has been widely defended and criticized, as it presents 
solutions to humanity’s problems, but also can pose quite numerous dangers. When dealing 
with transhumanism issues such as social inequality or political problems, in addition to the 
ones that we have already explored in this project, may arise. It is, therefore, of great 


importance to further analyze this possibility in the future. 


Also, we found that the control methods that are discussed by authors such as Bostrom does 
not emphasize on an ethical analysis, possibly because he does not take consciousness as a 
given. We believe that further investigation in the subject should also focus on ethical 
perspectives as an attempt of reducing potentially immoral actions by the people in charge of 


controlling the AI. 


These are just a few suggestions concerning further research about controlling an artificially 
created superintelligent being. However, other approaches can be used: not using sci-fi, basing 
on the feasibility of creating an AI, using different kinds of science fiction, etc. The issue we 
presented here should be studied from a variety of perspectives, as it can be an advance of such 


importance that it may change the world we live in forever. 
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